List extraction and visualization in web pages

Abstract

The present disclosure provides methods, apparatuses and computer program products for list extraction and visualization in web pages. At least one anchor element group in a target web page can be detected, the at least one anchor element group comprising a first anchor element group. Boundary detection can be performed on a plurality of anchor elements in the first anchor element group to obtain boundaries of a plurality of items respectively associated with the plurality of anchor elements, the plurality of items corresponding to a first original list in the target web page. The plurality of groups of representative metadata respectively corresponding to the plurality of items can be obtained from the target web page using the boundaries of the plurality of items. The plurality of groups of representative metadata can be visualized as a structured list.

Description

Background Technology

[0001] There are countless web pages on the internet, containing a wide variety of information. In some scenarios, internet users may need to find web pages of interest to obtain the information they desire. Search engine providers can offer search services to help users find these pages. For example, in response to a user's search query, the search service can return a search results page that includes information about the web pages related to the search query, such as web page links and snippets. Summary of the Invention

[0002] This invention is provided to introduce a set of concepts, which will be further described in the following detailed description. This invention is not intended to identify key or essential features of the protected subject matter, nor is it intended to limit the scope of the protected subject matter.

[0003] Embodiments of this disclosure provide methods, apparatus, and computer program products for list extraction and visualization in web pages. At least one group of anchor elements in a target web page can be detected, the at least one group of anchor elements including a first group of anchor elements. Boundary detection can be performed on multiple anchor elements in the first group of anchor elements to obtain boundaries of multiple items respectively associated with the multiple anchor elements, the multiple items corresponding to a first original list in the target web page. Using the boundaries of the multiple items, multiple sets of representative metadata corresponding to the multiple items can be obtained from the target web page. The multiple sets of representative metadata can be visualized as a structured list.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. Certain illustrative features of the one or more aspects are set forth in detail in the following specification and drawings. These features merely indicate various ways in which the principles of each aspect can be implemented, and this disclosure is intended to include all such aspects and their equivalents. Attached Figure Description

[0005] The following description will take into account several aspects disclosed, which are provided to illustrate rather than limit the aspects disclosed.

[0006] Figure 1 An example list webpage is shown.

[0007] Figure 2 An example list webpage is shown.

[0008] Figure 3 An existing exemplary search results page is shown.

[0009] Figure 4 An exemplary process for extracting and visualizing a list from a webpage according to an embodiment is shown.

[0010] Figure 5 An exemplary process for detecting anchor element groups according to an embodiment is shown.

[0011] Figure 6 An exemplary group of anchor elements according to an embodiment is shown.

[0012] Figure 7 An exemplary process for boundary detection according to an embodiment is shown.

[0013] Figures 8A to 8F An example of iterative boundary expansion according to an embodiment is shown.

[0014] Figures 9A to 9F An example of iterative boundary expansion according to an embodiment is shown.

[0015] Figure 10 An exemplary boundary detection result according to an embodiment is shown.

[0016] Figure 11 An exemplary process for determining the master list according to an embodiment is shown.

[0017] Figure 12 An exemplary process for obtaining representative metadata according to an embodiment is shown.

[0018] Figure 13 An exemplary search results page according to an embodiment is shown.

[0019] Figure 14 A flowchart illustrating an exemplary method for list extraction and visualization in a webpage, according to an embodiment, is shown.

[0020] Figure 15 An exemplary apparatus for list extraction and visualization in a webpage, according to an embodiment, is shown.

[0021] Figure 16 An exemplary apparatus for list extraction and visualization in a webpage, according to an embodiment, is shown. Detailed Implementation

[0022] This disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that this discussion of embodiments is merely intended to enable those skilled in the art to better understand and thus implement the embodiments of this disclosure, and is not intended to teach any limitation on the scope of this disclosure.

[0023] Existing search services typically extract specific text from the original webpage to form a text summary, which is then displayed on the search results page so that users can get a general understanding of the original webpage's content. Many list webpages exist online, where the main content is a list containing multiple items. For such list webpages, existing search services still simply extract specific text to form a text summary, and this summary may only contain text extracted from a specific item in the list, such as the first item. Therefore, when users see this text summary on the search results page, they only gain limited, localized information about the list webpage.

[0024] Embodiments of this disclosure propose performing list extraction and visualization on a webpage to extract list content from a target webpage and organize the list content into a structured form. In this document, the target webpage can be a list webpage. Embodiments of this disclosure can extract list content from a target webpage and visualize at least a portion of the extracted list content to form a list-based summary, i.e., a list summary. Compared to a text summary, a list summary can contain richer content about the original list on the target webpage, allowing users to gain a more comprehensive understanding of the original list from the list summary. Since the list summary itself is a structured list, embodiments of this disclosure can present information about the original list to users in a more user-friendly and intuitive way. Embodiments of this disclosure can present key or representative information about the items in the original list in the list summary, thereby providing comprehensive and concise information that users might expect. When the list summary is presented on a search results page, users can conveniently and comprehensively obtain the content of the corresponding target webpage without clicking on webpage links.

[0025] The original lists in the target web pages processed by the embodiments of this disclosure are not limited to those lists with HTML list tags, but can encompass any visually perceptible list. The "original list" involved in the embodiments of this disclosure is a visually perceptible list. A visually perceptible list can refer to, for example, a list containing multiple items with visually similar structures. In this document, "item" can refer to the components constituting the list, and can also be referred to as an object, entity, data record, etc. A visually perceptible list may or may not have HTML list tags; thus, a visually perceptible list can have any HTML tags without limitation. The embodiments of this disclosure are proposed at least for target web pages that include visually perceptible lists, and can process these target web pages at least from a visual perception perspective, rather than simply using HTML list tags. Accordingly, the embodiments of this disclosure can be applied to any target web page containing visual lists.

[0026] In one aspect, embodiments of this disclosure can identify a potential original list within a target webpage by detecting groups of anchor elements. The anchor elements within these groups do not necessarily have HTML list tags. Since anchor elements may contain representative information about items in the original list, detecting these groups of anchor elements will help in discovering the original list within the target webpage.

[0027] In one aspect, embodiments of this disclosure may perform boundary detection on multiple anchor elements in an anchor element group to determine the boundaries of multiple items in an original list corresponding to the anchor element group on a target webpage. Herein, determining the boundaries of an item may refer to determining which specific elements are included in the item, and consequently, which elements together form the item. Boundary detection may include iterative boundary expansion. For each anchor element, iterative boundary expansion can find elements that may be located within the same item as the anchor element, thereby defining the boundary of the item with the anchor element and the found elements. Boundary detection may also include a similarity check. A similarity check can be performed to determine whether multiple items determined by expansion from different anchor elements are indeed items in the same original list, for example, whether these items actually form an original list. At least through boundary detection according to embodiments of this disclosure, the original list in the target webpage and the individual items in that original list can be accurately identified.

[0028] In one aspect, if the target webpage comprises two or more original lists, embodiments of this disclosure can determine a dominant list from these original lists. Herein, a dominant list may refer to, for example, a list that occupies a prominent position on the webpage, presents the main content, etc. Preferably, by determining the dominant list and performing subsequent processing only on the dominant list, embodiments of this disclosure can include only information about the dominant list in the final generated structured list, thereby avoiding interference from information about lists that are not dominant lists.

[0029] In one aspect, embodiments of this disclosure can obtain multiple sets of representative metadata for different items in an original list from a target webpage. For example, multiple sets of representative metadata for these items can be obtained from the target webpage by utilizing at least the boundaries of multiple items in the original list. In some implementations, the multiple sets of representative metadata may be important, representative metadata selected from initial metadata in the target webpage by sorting.

[0030] In one aspect, embodiments of this disclosure can visualize multiple sets of representative metadata to form a structured list. This structured list can serve as, for example, a list summary of target web pages.

[0031] The embodiments of this disclosure can be applied to various application scenarios. For example, in a search service, embodiments of this disclosure can generate a structured list for a target webpage, in order to, for example, create a list summary for the target webpage. Accordingly, the search service can present the structured list generated as a list summary according to the embodiments of this disclosure on the search results page. It should be understood that the embodiments of this disclosure are not limited to application in search services, but can also be applied to any application scenario that requires list extraction and visualization of target webpages.

[0032] The target webpage processed in the embodiments of this disclosure can be a list of webpages from various websites, online services, etc. Figure 1 An exemplary list webpage is shown. List webpage 12 is an exemplary article on the web, which may be located on, for example, an academic website, an online Q&A community, etc. The article introduces the ten major festivals in China, such as "Spring Festival," "Mid-Autumn Festival," and "Dragon Boat Festival." The sections of the article that relate to the introduced festivals form a visually perceptible list 122. For example, the sections related to "Spring Festival," "Mid-Autumn Festival," and "Dragon Boat Festival" respectively form multiple items in list 122.

[0033] List page 14 originates from a webpage such as a book sales website or a reading exchange website. Assuming multiple options have been selected in the "Options" section on the left side of page 14, the right side of page 14 displays descriptions of four recommended books matching the selected options. Taking the first book as an example, its description might include, for example, a cover photo 144 and a text description 146. The descriptions of the four books form a visually perceptible list 142. For example, the description of each book forms an item in list 142.

[0034] List page 16 is a webpage from a review forum for the exemplary topic "Restaurant X," which includes multiple user discussion threads about "Restaurant X." For example, page 16 includes multiple display areas for users Tom, David, Jane, etc. Taking user Tom as an example, the display area for Tom includes, for example, Tom's avatar, Tom's name, the posting time of Tom's review, and the specific content of Tom's review. The display areas for Tom, the display areas for David, the display areas for Jane, etc., form a visually perceptible list 162, and these display areas respectively form multiple items in list 162.

[0035] Figure 2An exemplary list webpage 20 is shown. Webpage 20 may originate from, for example, an online shopping website. Online shopping websites typically generate or provide numerous webpages that include lists, such as best-selling webpages, most popular product webpages, product category webpages, webpages containing products searched by the user, etc. Webpage 20 may be a webpage used to present, for example, mobile phones that meet certain criteria. Assuming that multiple options have been selected in the "Options" bar on the left side of webpage 20, the right side of webpage 20 displays information about multiple mobile phones that match the selected options. For example, area 22 displays information about the mobile phone "M Mobile Phone A4", including, for example, a picture of the phone 222, a description of the phone "M Mobile Phone A4, 6.5-inch, 256G, Black", a 5-star rating for the phone, the number of reviews about the phone "25,900 reviews", and the price of the phone "5,500 RMB", etc. Similarly, area 24 presents introductory information for mobile phone "M Mobile Phone A3" including at least image 242, area 26 presents introductory information for mobile phone "M Mobile Phone A2" including at least image 262, and so on. This introductory information for these mobile phones forms a visually perceptible list 202, where the introductory information for each mobile phone forms an item in list 202. Furthermore, webpage 20 also presents recommendations for related products in area 28, such as introductory information for a first related product including at least image 282, introductory information for a second related product including at least image 284, introductory information for a third related product including at least image 286, and so on. This introductory information for these related products forms a visually perceptible list 204, where the introductory information for each related product forms an item in list 204.

[0036] It should be understood that the embodiments disclosed herein are not limited to those described above. Figure 1 and Figure 2 The exemplary list webpage shown is not limited to any one of the many other types of list webpages from various other websites, online services, etc., such as list webpages from forum websites on topics in various fields, list webpages from product review websites, list pages from news websites, list webpages from hotel or flight booking websites, and so on.

[0037] Figure 3 An existing exemplary search results page 300 is shown. Search results page 300 may be presented to a user within a search service provided by a general search engine provider. Assume a user has already entered the query "M mobile phone" into search box 310, indicating that they want web search results about M mobile phones. Search results area 320 in search results page 300 includes multiple web search results. For example, area 330 shows results for... Figure 2The search results for webpage 20 are shown in area 330. As shown in area 330, the search results for webpage 20 include a text summary: "M Phone A4, 6.5-inch, 256GB, Black, 5 stars, 25,900 reviews, 5,500 RMB". This text summary is generated solely from the information about "M Phone A4" on webpage 20. Based on this text summary in area 330, users can only learn limited information about webpage 20; for example, they can only learn about the phone "M Phone A4," but cannot obtain any information about the other phones in list 202 on webpage 20. Furthermore, such a text summary lacks intuitiveness and readability.

[0038] Figure 4 An exemplary process 400 for list extraction and visualization from a webpage according to an embodiment is illustrated. Process 400 can be executed to extract and visualize a list from a raw list in a target webpage 402 to generate a structured list 404. The target webpage 402 can be a list webpage containing lists, and the lists in the target webpage 402 can be referred to herein as the raw list. If the target webpage 402 includes two or more lists, process 400 can generate the structured list 404 from the main list in the target webpage 402.

[0039] At point 410, at least one group of anchor elements in the target webpage 404 can be detected. Each group of anchor elements may include one or more anchor elements, and each group of anchor elements may correspond to a possible original list. For example, if the target webpage 404 includes two or more original lists, then at point 410, two or more groups of anchor elements corresponding to each of these original lists may be detected. In one implementation, multiple anchor elements in the target webpage 404 may be identified first, and then the multiple anchor elements may be clustered into at least one group of anchor elements.

[0040] At 420, for each anchor element group, boundary detection can be performed on multiple anchor elements within that anchor element group to obtain the boundaries of multiple items associated with each of the multiple anchor elements. These items can form the original list in the target webpage 404 corresponding to that anchor element group. Boundary detection can include, for example, iterative boundary expansion, similarity checks, etc., to accurately identify at least one original list in the target webpage 404 and each item in each original list.

[0041] At 430, optionally, if the previous steps have determined that the target webpage 404 includes two or more original lists, the main list can be determined from these original lists. In one implementation, the main list can be determined using at least the visual characteristics of these original lists.

[0042] At 440, for the original list in the target webpage 404, multiple sets of representative metadata corresponding to each of the multiple items in the original list can be obtained from the target webpage 404, utilizing at least the boundaries of the multiple items in the original list. Optionally, the acquisition of representative metadata at 440 can be performed on the main list in the target webpage 404. In this document, representative metadata can refer to data contained in the original list and to be presented in the structured list 404, such as images, text, etc. In one implementation, for each item, an initial set of metadata can be obtained from the target webpage 404 first, and then a set of representative metadata corresponding to that item can be selected from this initial set of metadata to be presented in the structured list 404.

[0043] At position 450, the multiple sets of representative metadata obtained at position 440 can be visualized as a structured list 404. In one implementation, the structured list 404 can be formed using the multiple sets of representative metadata according to a predetermined format or layout. The structured list 404 is a simplified version of the original list in the target webpage 402, but it still contains enough information to allow the user to intuitively and comprehensively understand the main content of the original list. The structured list 404 can serve as a list summary, for example, of the original list.

[0044] It should be understood that all steps in process 400 and their order are exemplary, and embodiments of this disclosure will also cover modifications to process 400 in any manner. For example, although process 400 includes the step of determining the main list at 430, this step may be omitted if the target webpage 402 includes only a single original list. For example, although in Figure 4 The diagram shows that step 430 is performed before step 440, but step 430 can also be performed after step 440. In this case, multiple sets of representative metadata for each original list can be obtained first through step 440, and then, after the master list is determined through step 430, only the multiple sets of representative metadata for the master list can be provided to step 450.

[0045] Figure 5 An exemplary process 500 for anchor element group detection according to an embodiment is shown. Process 500 is Figure 4 An exemplary implementation of step 410 in the example.

[0046] At point 510, multiple anchor elements can be identified from the target webpage 502. For example, these anchor elements can be identified from the HTML source file of the target webpage 502. Each item in the original list may include multiple HTML elements, and the anchor element can be the most representative HTML element that is most helpful in identifying the entire item. The anchor element identified at point 510 can also be referred to as the identification anchor element. In one implementation, anchor element constraints can be predefined, and multiple HTML elements in the target webpage 502 that satisfy the anchor element constraints can be identified as multiple identification anchor elements. For example, anchor element constraints may include at least one of the following: the HTML element has an image tag; the HTML element has a title tag; the HTML element represents a date; etc. In one case, each item in the original list may have a corresponding image, therefore, the HTML source file has an image tag, for example... HTML elements such as tags can be used as anchor elements to help identify the corresponding items. In one case, each item in the original list may have a corresponding title; therefore, the HTML source file contains title tags, such as... <h1> Label,< / h1> <h2>HTML elements such as tags can serve as anchor elements to help identify the corresponding items. In one case, each item in the original list may have a date, such as a post date. Therefore, HTML elements in the HTML source file containing strings representing dates can serve as anchor elements to help identify the corresponding items. In this case, various techniques, such as regular expression matching, can be used to identify the date strings in the HTML source file. It should be understood that the embodiments of this disclosure are not limited to the exemplary anchor element constraints described above, but can cover any other types of anchor element constraints.

[0047] At step 520, the set of properties for each of the multiple identified anchor elements identified at step 510 can be extracted from the target webpage 502. The property set of an identified anchor element may include one or more inherent properties of that anchor element, such as its HTML tag attribute, Cascading Style Sheets (CSS) class, XPath information, etc. The HTML tag attribute can indicate the HTML tag type of the identified anchor element. The CSS class can indicate which CSS classes the identified anchor element has. The XPath information can indicate the positioning information, node information, etc., of the identified anchor element, which may be obtained, for example, from the Document Object Model (DOM) tree corresponding to the HTML source file. It should be understood that the embodiments of this disclosure are not limited to the exemplary properties of the identified anchor elements described above, but can cover any other type of property. Through step 520, multiple property sets corresponding to the multiple identified anchor elements can be obtained.

[0048] At point 530, the identification anchor elements can be clustered into at least one anchor element group 504 based on multiple attribute sets of multiple identification anchor elements. Each identification anchor element can be characterized by a corresponding attribute set, and the multiple attribute sets of multiple identification anchor elements can be provided as input to a pre-trained clustering model. The clustering model is trained to cluster multiple anchor elements into at least one anchor element group based on the attribute sets. For example, identification anchor elements with similar attributes will be clustered into the same anchor element group. Each anchor element group includes multiple anchor elements with similar attributes and can correspond to a possible original list, wherein these anchor elements can be associated with different items in the possible original list.

[0049] It should be understood that all steps in process 500 are exemplary, and embodiments of this disclosure will also cover modifications to process 500 in any manner. For example, process 500 may employ any combination of various anchor element constraints, a set of attributes containing any combination of various attributes, etc.

[0050] Figure 6 An exemplary group of anchor elements according to an embodiment is shown. Figure 6 In the middle, assuming that for Figure 2 The webpage 20 detected a first anchor element group and a second anchor element group. The first anchor element group may include multiple anchor elements, which correspond to... Figure 2 Images 222, 242, and 262 in the original list 202 are clustered into a first anchor element group due to having similar attributes. The second anchor element group may include multiple anchor elements, which correspond to... Figure 2 Images 282, 284, 286, etc. in the original list 204, wherein images 282, 284, and 286 can be clustered into the second anchor element group due to having similar attributes.

[0051] It should be understood that, despite Figure 6 The illustration shows a group of anchor elements detected based on the anchor element constraint "html element has image tag", but embodiments of this disclosure can also detect groups of anchor elements based on other types of anchor element constraints. For example, for Figure 1 Webpage 12 can detect anchor element groups formed by titles such as "Spring Festival", "Mid-Autumn Festival", and "Dragon Boat Festival" based on the anchor element constraint of "html element has title tag". For example, for Figure 1 Webpage 16 can detect the anchor element group formed by Tom's display area "2021-10-05", David's display area "2021-10-05", Jane's display area "2021-10-06", etc., based on the anchor element constraint of "html element represents date".

[0052] Figure 7 An exemplary process 700 for boundary detection according to an embodiment is shown. Process 700 is... Figure 4 An exemplary implementation of step 420 is described below. Process 700 can be used to perform boundary detection on an exemplary group of anchor elements 702 to obtain the boundaries of multiple items associated with the multiple anchor elements in the group 702, thereby identifying the original list 704 corresponding to the anchor element group 702 in the target webpage and each item in that original list 704. Process 700 can be performed at least based on the DOM tree corresponding to the target webpage.

[0053] At point 710, iterative boundary expansion can be performed on each anchor element in anchor element group 702 to find elements that may be located within the same item as that anchor element. In one implementation, iterative boundary expansion can be performed synchronously, starting from multiple anchor elements in anchor element group 702, based on the DOM tree corresponding to the target webpage. Each anchor element can serve as a starting point, and through iterative boundary expansion, multiple other elements can be sequentially identified and expanded in the DOM tree from that anchor element. The anchor element, together with the identified other elements, forms a tree that represents an item, and is therefore also called an item tree. Multiple nodes in the item tree can each correspond to multiple elements, such as the anchor element and the elements identified through iterative boundary expansion. Each iteration can expand to the next node, and the next node can be included in the item tree. Multiple iterations form a corresponding expansion path. Through iterative boundary expansion at point 710, multiple item trees originating from multiple anchor elements in anchor element group 702 can be obtained. The multiple item trees each define the boundaries of multiple items.

[0054] Iterative boundary expansion can include various types of expansion, such as sibling node expansion and parent node expansion. Sibling node expansion can be performed to expand from the current node to its sibling nodes in the DOM tree corresponding to the target webpage. In one case, if the current node has multiple sibling nodes belonging to the same parent node, expansion can be performed sequentially from the current node outwards to these sibling nodes. In another case, sibling node expansion can adopt a predetermined expansion direction, such as expanding to the right, expanding to the left, alternating between expanding to the right and left, changing to expanding to the left after multiple right expansions or meeting predetermined conditions, changing to expanding to the right after multiple left expansions or meeting predetermined conditions, and so on. Parent node expansion can be performed to expand to the parent node of the current node after all its sibling nodes have been included in the same item tree, and to include that parent node in the item tree. After expanding to the parent node, sibling node expansion can continue to be performed on the parent node, for example, expanding to the parent node's sibling nodes. This process can be repeated iteratively to expand upwards to higher-level nodes. Furthermore, if a node is included in the project tree through iterative boundary expansion and that node has its own subordinate nodes, such as child nodes, grandchild nodes, etc., then all subordinate nodes of that node can also be included in the project tree.

[0055] Iterative boundary expansion can be performed synchronously between different project trees corresponding to different anchor elements. For example, in each iteration step, a sibling node expansion or parent node expansion is performed synchronously in these project trees. In one case, for example, during a sibling node expansion in a certain step, if a project tree S currently has no sibling nodes to expand, while other project trees have sibling nodes to expand, then while performing the sibling node expansion for the other project trees in the current step, the expansion of project tree S can be paused once in the current step.

[0056] According to embodiments of this disclosure, the boundaries of each item can be expanded as much as possible, for example, by iteratively expanding the boundaries so that each item includes as many elements as possible. However, there should be no content overlap between different items; for example, the same element or content should not be included in different items. Furthermore, the structures of different items should be similar; for example, different items should have at least a predetermined proportion of similar elements or nodes.

[0057] Content overlap between two projects may be caused by overlapping nodes in the two project trees corresponding to these two projects; for example, a node may be shared by both project trees. Therefore, content overlap between different projects can be avoided by detecting node overlap during the iterative boundary expansion at 710.

[0058] At 720, it can be determined whether the iteration of the current step in the iterative boundary expansion results in node overlap between at least two project trees. For example, whether the iteration of the current step results in the inclusion of one or more identical nodes in at least two project trees simultaneously. The node overlap determination at 720 can be performed synchronously with the iterative boundary expansion at 710, for example, determining whether node overlap exists after each iteration step.

[0059] If it is determined at 720 that the iteration of the current step has not resulted in node overlap, then process 700 can return to 710 and continue with the iterative boundary expansion.

[0060] If it is determined at 720 that the current step's iteration results in node overlap, then the iteration boundary expansion is stopped at 730, and the nodes determined by the current step's iteration are excluded from each item tree. For example, this causes each item tree to revert or reset to its state at the previous iteration before the current step's iteration.

[0061] By performing steps 720 and 730, embodiments of this disclosure can avoid node overlap in the resulting multiple project trees, thereby avoiding content overlap between different projects.

[0062] To determine whether different projects have similar structures, process 700 may perform a similarity test on multiple project trees. In one case, the similarity test may be performed in response to determining that the number of nodes in at least one of the multiple project trees exceeds a node count threshold. In one aspect, for example, the at least one project tree may be a predetermined number or predetermined proportion of project trees among the multiple project trees, thus the execution of the similarity test may require that the number of nodes in each project tree within the predetermined number or predetermined proportion of project trees among the multiple project trees exceeds the node count threshold. In another aspect, for example, the execution of the similarity test may require that the iterative boundary expansion at 710 has been iterated for a predetermined number of steps, i.e., each project has contained a predetermined number of elements or each project tree has contained a predetermined number of nodes. In one case, the similarity test may be performed synchronously with the iterative boundary expansion at 710, for example, after each iteration step. In one scenario, a similarity check can be performed after a predetermined number of iterations, for example, after a predetermined number of elements are added to each item or after a predetermined number of nodes are added to each item tree. Embodiments of this disclosure are not limited to the exemplary timing of performing similarity checks as described above.

[0063] At point 740, tree similarity between any two item trees in a plurality of item trees can be calculated. Embodiments of this disclosure are not limited to any particular technique for calculating tree similarity. Preferably, embodiments of this disclosure propose a tree similarity calculation method that improves upon existing simple tree matching algorithms, the proposed method utilizing at least CSS similarity-based weights and / or minimum depth levels to calculate tree similarity.

[0064] In one implementation, embodiments of this disclosure may calculate tree similarity using at least a matching weight calculated based on the CSS similarity between the root nodes of two item trees. In a webpage, the style rendered by CSS is crucial information indicating the page layout. Therefore, by calculating a matching weight based on the CSS similarity between the root nodes of two item trees and using that matching weight to calculate the tree similarity between the two item trees, the accuracy of tree similarity calculation can be effectively improved. The matching weight may be calculated based, for example, the respective CSS categories of the two root nodes.

[0065] In one implementation, embodiments of this disclosure can utilize nodes within a minimum depth level in two item trees to calculate tree similarity. Here, the minimum depth level can be defined such that the number of visible nodes within the minimum depth level of an item tree reaches a predetermined proportion, such as 80%, or any other proportion of the total number of visible nodes in that item tree. Alternatively, the minimum depth level can also be defined such that the number of visible nodes within levels of the item tree below the minimum depth level fails to reach a predetermined proportion of the total number of visible nodes in that item tree. Here, visible nodes can refer to visually visible nodes in a webpage, such as nodes displaying images, nodes displaying text, etc., and therefore, visible nodes are more important for determining the structural similarity between item trees compared to other nodes. An item tree may have multiple levels; for example, assuming the root node of the item tree is at a depth of 0, then the child nodes of that root node are at a depth of 1, and so on. The deeper the level, the less contribution it makes to determining the structural similarity between the two trees. Therefore, embodiments of this disclosure propose using only a portion of the levels of the item tree, rather than all levels, to calculate tree similarity, thereby effectively improving computational efficiency and saving computational resources. Tree similarity can be calculated using the minimum depth level and those levels lower than the minimum depth level. For example, assuming the minimum depth level is 3, tree similarity can be calculated using levels with depths of 0, 1, 2, and 3. Since the minimum depth level is determined by considering at least the number of visible nodes—for example, the number of visible nodes within the minimum depth level should not be less than a predetermined proportion of the total number of visible nodes in the item tree—an appropriately set predetermined proportion can ensure accurate tree similarity calculation even if levels greater than the minimum depth level are not considered in the tree similarity calculation. The predetermined proportion can have any value preset according to actual application requirements. It should be understood that although the above discussion uses nodes within the minimum depth level of two item trees to calculate tree similarity, embodiments of this disclosure are not limited to this, but can alternatively use nodes within all levels of two item trees to calculate tree similarity.

[0066] Suppose T and T' are two project trees. Root(T) represents the root node of tree T, and Root(T') represents the root node of tree T'. It should be understood that if T and T' do not have actual root nodes, virtual root nodes can be set for T and T' respectively, and these two virtual root nodes can have the same attribute configuration. For each of T and T', L0, L1, ..., L n Let L represent the sets of subtrees at depths of 0, 1, ..., n. i1 ,L i2 ,…,L ik Let L represent the k subtrees at a level depth of i, that is, the set of subtrees L. i The subtree in.

[0067] Let css1 represent the set of CSS categories possessed by Root(T), where Root(T) may have 0, 1, or any other number of CSS categories. Let css2 represent the set of CSS categories possessed by Root(T'), where Root(T') may have 0, 1, or any other number of CSS categories. In one implementation, the Jaccard coefficient can be used to calculate the matching weight between T and T'. For example, the matching weight between T and T' can be calculated as:

[0068]

[0069] MatchWeight(·) is a function that calculates the match weight. |css1| represents the number of CSS categories contained in css1, |css2| represents the number of CSS categories contained in css2, and |css1∩css2| represents the number of CSS categories that are commonly contained in css1 and css2.

[0070] The tree similarity between T and T' can be calculated based on, for example, the process shown in Table 1 below.

[0071]

[0072] Table 1

[0073] In step 1.1, the minimum depth level MinDepth can be determined. In step 1.2, a similarity check function SimilarityCheck(·) is defined to calculate the similarity measure of T and T' at the current layer. This function can include the processing in subsequent steps 1.3 to 1.18. In steps 1.3 and 1.4, if it is determined that Root(T) and Root(T') have different html tags, the result of SimilarityCheck(·) is 0. In steps 1.5 and 1.6, if it is determined that the current layer is greater than the minimum depth level MinDepth, the result of SimilarityCheck(·) is 0, thus avoiding tree similarity calculation when the current layer is greater than the minimum depth level. In step 1.8, m is used to represent the set L of subtrees in T corresponding to depth layer+1. layer+1 The subtrees in T'. In step 1.9, n is used to represent the set of subtrees L' in T' corresponding to depth layer+1. layer+1 The subtrees in T. In the process of Table 1, a similarity function M[i,j] is defined, which represents the maximum similarity between the first i subtrees in T and the first j subtrees in T'. In steps 1.10 and 1.11, M[i,0] and M[0,j] are initialized to 0 respectively. In step 1.12, the subtrees in the subtree set m of T that will be traversed are defined. In step 1.13, the subtrees in the subtree set n of T' that will be traversed are defined. In steps 1.14 and 1.15, the similarity can be calculated using the similarity function M[i,j]. For example, dynamic programming or other techniques can be used to perform the calculations in steps 1.14 and 1.15. M[i,j] will obtain the best similarity from three candidates, namely M[i,j-1], M[i-1,j], and M[i-1,j-1]+W[i,j]. W[i,j] will recursively compute the i-th subtree T in layer+1 of T. i The j-th subtree T' in T' j The similarity between them. Thus, the entire tree structure can be considered, not just the root node. At step 1.18, the result of SimilarityCheck(·) can be returned, which is represented as MatchWeight(Root(T),Root(T'))*(M[m,n]+1), where M[m,n] is the best similarity of the subtrees of T and T', and "1" represents the root node. It should be understood that the result returned at step 1.18 may indicate, for example, the number of similar nodes. At step 1.19, the final tree similarity can be further calculated using all nodes with a depth no greater than the minimum depth level, where TreeSimilarity(·) is the tree similarity function, |T| is the number of nodes in T with a depth no greater than the minimum depth level, and |T′| is the number of nodes in T' with a depth no greater than the minimum depth level. It should be understood that all steps in Table 1 are exemplary, and embodiments of this disclosure will also cover any modifications to these steps.

[0074] After calculating the tree similarity between any two project trees in the multiple project trees through step 740, the multiple project trees can be divided into at least one tree set at 745 using at least a similarity threshold. Each tree set in the at least one tree set may include at least one project tree, and at least one project tree in the same tree set has a tree similarity to each other that is not less than the similarity threshold. Through step 745, project trees with high similarity to each other can be grouped into the same tree set. At 750, it can be determined whether the number of project trees in the tree set containing the most project trees in the at least one tree set is less than the tree number threshold. The tree set containing the most project trees can be used as the target tree set to determine whether the iteration should be stopped. The tree number threshold may have a preset value, which can be used, for example, to ensure that most of the project trees in the multiple project trees are included in the target tree set. If it is determined at 750 that the number of project trees in the target tree set is not less than the tree number threshold, then process 700 can return to 710 and continue to perform iterative boundary expansion. If it is determined at 750 that the number of item trees in the target tree set is less than a tree number threshold, then the iterative boundary expansion can be stopped at 760, and nodes determined by a predetermined number of previous iterations can be excluded from the multiple item trees. In one implementation, the predetermined number of previous iterations can be determined by excluding nodes determined by a predetermined number of previous iterations from the multiple item trees, ensuring that the number of item trees in the target tree set obtained, for example, for the updated multiple item trees via steps 740 and 745, is not less than a tree number threshold.

[0075] By performing steps 740, 745, 750, and 760, embodiments of this disclosure can perform similarity checks on multiple item trees. Similarity checks help ensure that the resulting item trees have structural similarity. For example, when multiple item trees from a target tree set are provided as the result of an iterative expansion, these item trees in the iterative expansion result will have high similarity to each other, thereby ensuring that different items have similar structures.

[0076] Optionally, according to process 700, further iterative boundary expansion can be performed at 770 to attempt to find multiple better project trees. In one implementation, after performing step 730, if it is determined that the iteration of the current step is a sibling node expansion, further iterative boundary expansion can be performed on the multiple project trees in the opposite direction to the iteration of the current step. For example, for each project tree, if the iteration of the current step is to expand to the right to a sibling node, an attempt can be made to expand to the left to a different sibling node. In another implementation, after performing step 760, the multiple project trees can first be reset to the state at the iteration of a predetermined previous step. The iteration of the predetermined previous step can be determined, for example, using the predetermined number of iterations of the previous steps involved at 760. For example, if the predetermined number of iterations of the previous steps is the iteration of the previous two steps, then the iteration of the predetermined previous step can be the iteration of the previous third step. Then, if it is determined that the iteration of the next step after the iteration of the predetermined previous step is a sibling node expansion, further iterative boundary expansion can be attempted on the multiple project trees in the direction opposite to the direction of the next step iteration. For example, for each project tree, if the project tree was reset to the state at the previous third step iteration, and the previous second step iteration was an expansion to the right to a sibling node, then an attempt can be made to expand to the left to a different sibling node.

[0077] It should be understood that, although not shown, process 700 may also include performing steps 720 to 730 and / or steps 740 to 760 for further iterative boundary expansion at 770 in order to ensure that the obtained multiple project trees do not have overlapping nodes and have similar structures, thereby ensuring that different projects do not have overlapping content and have similar structures.

[0078] Through process 700, multiple item trees can be obtained, each originating from multiple anchor elements in anchor element group 702. These item trees define the boundaries of multiple corresponding items, thereby ultimately identifying the original list 704 formed by these items.

[0079] It should be understood that all steps in process 700 are exemplary, and embodiments of this disclosure will also cover modifications to process 700 in any manner. For example, the processes in process 700 concerning determining whether node overlap occurs and the processes concerning performing similarity checks are both optional; either or both of these processes may be included in process 700, or either or both of these processes may be omitted from process 700. Furthermore, for example, process 700 may also provide only multiple item trees from the target tree set as an iterative expansion result and utilize these item trees to form the original list 704.

[0080] Figures 8A to 8F An example of iterative boundary expansion according to an embodiment is shown. Figures 8A to 8F In the example, an exemplary process of iterative boundary expansion is shown in the exemplary DOM tree corresponding to the target webpage.

[0081] The DOM tree may include multiple nodes, such as nodes 801 to 826 and others not shown. Symbols such as "Div", "A", "Span", "P", and "Img" displayed in the boxes representing nodes indicate the corresponding HTML tags. Furthermore, embodiments of this disclosure propose setting a "Text" tag for text strings appearing in the HTML source file, even though such an HTML tag does not exist, and these text strings can also be nodes in the DOM tree, such as nodes 819, 820, etc. These text strings may be renderable visible elements; therefore, setting the Text tag and corresponding node for these text strings will help to more accurately determine item boundaries. It should be understood that, for illustrative purposes, Figures 8A to 8F Only a few exemplary node labels are shown; in practical applications, any other types of node labels may exist, and the embodiments of this disclosure are not limited to what specific labels the nodes in the DOM tree may have. Furthermore, in Figures 8A to 8F In the middle, nodes included in the project tree through iterative boundary expansion are highlighted with shading, and arrows indicate the expansion path of iterative boundary expansion.

[0082] exist Figure 8A In this example, we assume that nodes 818, 821, and 824 have been identified as anchor elements with the Img (image) label. Then, iterative boundary expansion can be performed synchronously starting from each of these nodes to obtain item trees originating from each node. In the following text, the item tree originating from node 818 is referred to as the first item tree, the item tree originating from node 821 as the second item tree, and the item tree originating from node 824 as the third item tree.

[0083] Figure 8B The iteration of step 1 is shown. Since nodes 818, 821, and 824 have no sibling nodes, parent node expansion is performed in the iteration of step 1. For example, in the first item tree, expansion is performed from node 818 to its parent node 806; in the second item tree, expansion is performed from node 821 to its parent node 810; and in the third item tree, expansion is performed from node 824 to its parent node 814.

[0084] Figure 8C The iterations of steps 2 through 4 are illustrated, in which sibling node expansion is performed. Taking the first item tree as an example, node 806, determined by the iteration of step 1, is the current node, which has sibling nodes 807, 808, and 809. Therefore, the iterations of steps 2 through 4 will expand to the right sequentially as shown by the arrows, extending to nodes 807, 808, and 809. Similarly, in the second item tree, the iterations of steps 2 through 4 will expand to the right sequentially as shown by the arrows, extending to nodes 811, 812, and 813; and in the third item tree, the iterations of steps 2 through 4 will expand to the right sequentially as shown by the arrows, extending to nodes 815, 816, and 817. Furthermore, since node 808 has a child node 819 and node 809 has a child node 820, nodes 819 and 820 can also be included in the first item tree. Similarly, child nodes 822 of node 812 and child nodes 823 of node 813 can be included in the second item tree, and child nodes 825 of node 816 and child nodes 826 of node 817 can be included in the third item tree.

[0085] Figure 8D The iteration of step 5 is shown, in which parent node expansion is performed. Taking the first item tree as an example, since all sibling nodes 807, 808, and 809 of node 806 have already been included in the first item tree through the iterations of steps 2 through 4, the iteration of step 5 will extend to the parent node 803 of nodes 806 through 809, as shown by the arrow. Similarly, in the second item tree, the iteration of step 5 will extend to node 804, as shown by the arrow; and in the third item tree, the iteration of step 5 will extend to node 805, as shown by the arrow.

[0086] Figure 8E The iteration of step 6 is illustrated, in which parent node expansion is performed. Taking the first item tree as an example, node 803, determined by the iteration of step 5, has no sibling nodes; therefore, the iteration of step 6 will expand to the parent node 802 of node 803, as shown by the arrow. Similarly, in the second item tree, the iteration of step 6 will expand to node 802, as shown by the arrow; and in the third item tree, the iteration of step 6 will expand to node 802, as shown by the arrow.

[0087] Through the iterations in step 6, node 802 will be included in the first item tree, the second item tree, and the third item tree simultaneously, resulting in node overlap. Therefore, the iterative boundary expansion will stop, and node 802, determined through the iterations in step 6, will be excluded from the first item tree, the second item tree, and the third item tree, respectively. Figure 8F The final obtained first item tree 830, second item tree 840, and third item tree 850 are shown in dashed boxes. The first item tree 830, second item tree 840, and third item tree 850 correspond to the first, second, and third items in the original list on the target webpage, respectively. Thus, through... Figures 8A to 8F The iterative boundary expansion in the code can identify the original list and the items in that original list within the target webpage. It should be understood that it can also be used for... Figure 8F The final project tree shown above is executed in combination. Figure 7 The described similarity test. Furthermore, it should be noted that, as Figure 8F As shown, each project tree has its own root node. For example, the first project tree 830, the second project tree 840, and the third project tree 850 have their own root nodes 803, 804, and 805, respectively. Thus, the boundary of each project tree can actually be indicated by the html tag of the root node. For example, the boundary of the first project tree can be indicated by the "Div" tag of the root node 803.

[0088] Figures 9A to 9F An example of iterative boundary expansion according to an embodiment is shown. Figures 9A to 9F Examples of iterative boundary expansion in different ways are shown in an exemplary DOM tree corresponding to the target webpage. This DOM tree may include multiple nodes, such as nodes 901 to 929, and other nodes not shown.

[0089] exist Figure 9A In this example, we assume that nodes 919, 923, and 927 have been identified as anchor elements with the Img (image) label. Then, iterative boundary expansion can be performed synchronously starting from each of these nodes to obtain item trees originating from each node. In the following text, the item tree originating from node 919 is referred to as the first item tree, the item tree originating from node 923 as the second item tree, and the item tree originating from node 927 as the third item tree.

[0090] Figure 9B The iteration of step 1 is shown. Since nodes 919, 923, and 927 have no sibling nodes, parent node expansion is performed in the iteration of step 1. For example, in the first item tree, node 919 is expanded to its parent node 904; in the second item tree, node 923 is expanded to its parent node 909; and in the third item tree, node 927 is expanded to its parent node 914.

[0091] Figure 9C It shows in Figure 9B Based on the iteration of step 1, the project tree is ultimately obtained by performing subsequent iteration boundary expansion in an extended manner. For example... Figure 9C As shown, the iterations from steps 2 to 5 will perform sibling node expansion sequentially to the left, as indicated by the arrows. In the first item tree, the iteration from step 2 will expand to the left, as indicated by the arrows, towards node 903, and since there are no other sibling nodes, the iterations from steps 3 to 5 will pause in the first item tree. In the second item tree, the iterations from steps 2 to 5 will expand to the left, as indicated by the arrows, towards nodes 908, 907, 906, and 905, respectively. In the third item tree, the iterations from steps 2 to 5 will expand to the left, as indicated by the arrows, towards nodes 913, 912, 911, and 910, respectively. Since node 902 is the common parent node of the first, second, and third item trees, to avoid node overlap, the final first, second, and third item trees do not include node 902. Figure 9C The final obtained first project tree 932, the final obtained second project tree 934, and the final obtained third project tree 936 are shown in dashed boxes. It should be noted that nodes 915, 916, 917, 928, and 929 are not included in any project tree. Furthermore, it is assumed that the above combination was further performed on these final obtained project trees. Figure 7 The described similarity test revealed that although the second item tree 934 and the third item tree 936 had a high similarity, the tree similarity between the first item tree 932 and both the second and third item trees 934 and 936 was below the similarity threshold. Therefore, the first item tree 932 can be considered an unacceptable item tree and is discarded accordingly. Figure 9C The expansion method actually only outputs two project trees, 934 and 936.

[0092] Figure 9D It shows in Figure 9B Based on the iteration of step 1, the project tree is ultimately obtained by performing subsequent iteration boundary expansion in a different way. For example... Figure 9D As shown, the iterations of steps 2 through 5 will perform sibling node expansion sequentially to the right, as indicated by the arrows. In the first item tree, the iterations of steps 2 through 5 will expand to the right sequentially to nodes 905, 906, 907, and 908, as indicated by the arrows. In the second item tree, the iterations of steps 2 through 5 will expand to the right sequentially to nodes 910, 911, 912, and 913, as indicated by the arrows. In the third item tree, the iterations of steps 2 through 4 will expand to the right sequentially to nodes 915, 916, and 917, as indicated by the arrows, and since there are no further sibling nodes, the iteration of step 5 will be paused in the third item tree. Figure 9D The final obtained first project tree 942, the final obtained second project tree 944, and the final obtained third project tree 946 are shown in dashed boxes. It should be noted that nodes 903 and 918 are not included in any project tree. Furthermore, it is assumed that the above combination was further performed on these final obtained project trees. Figure 7 The described similarity test was performed, and it was found that the tree similarity between these item trees was not lower than the similarity threshold. Accordingly, Figure 9D The expansion method actually outputs all three project trees: 942, 944, and 946.

[0093] According to embodiments of this disclosure, in order to find a better project tree, one can, for example... Figure 7 Step 770 in the code is used to perform further iterative boundary expansion. Assume that... Figure 9D The project tree shown is reset to the state at the iteration in step 4, as follows: Figure 9E As shown. In Figure 9E In the first item tree, the current expansion path includes nodes 919, 904, 905, 906, and 907 as shown by the arrows. The current expansion path of the second item tree includes nodes 923, 909, 910, 911, and 912 as shown by the arrows. The current expansion path of the third item tree includes nodes 927, 914, 915, 916, and 917 as shown by the arrows. Figure 9D The fifth iteration in the previous step expands the sibling node to the right, unlike this. Figure 9F The iteration in step 5 can attempt to expand the sibling node to the left, that is, by using... Figure 9D The iteration direction of step 5 in the process is opposite to that of the previous step. Accordingly, in Figure 9F In the fifth iteration of the process, the expansion path of the first item tree will further include node 903, as shown by the arrow; the expansion path of the second item tree will further include node 908, as shown by the arrow; and the expansion path of the third item tree will further include node 913, as shown by the arrow. Figure 9F The dashed boxes in the diagram illustrate the final obtained first project tree 952, the final obtained second project tree 954, and the final obtained third project tree 956. Furthermore, it is assumed that the above combination was further performed on these final obtained project trees. Figure 7 The described similarity test was performed, and it was found that the tree similarity between these item trees was not lower than the similarity threshold. Accordingly, Figure 9F The expansion method actually outputs all three project trees: 952, 954, and 956.

[0094] In embodiments of this disclosure, the execution of iterative boundary expansion can follow predetermined criteria, such as obtaining as many project trees as possible, having as many nodes in each project tree as possible, and having high tree similarity between different project trees as possible, etc. Through Figure 9C , Figure 9D and Figure 9F A comparison between them shows that Figure 9D and Figure 9F The expansion method will be superior to Figure 9C The way to extend, because Figure 9D and Figure 9F The expansion method can output a larger number of item trees. Furthermore, Figure 9F The expansion method will be superior to Figure 9D The reason for this extension is: Figure 9F The expansion method can include more nodes (e.g., node 903 and node 918) into the project tree; and through Figure 9F The expansion method results in higher tree similarity between item trees. For example, the tree similarity between the third item tree 956 and the first item tree 952 and the second item tree 954 will be higher than the tree similarity between the third item tree 946 and the first item tree 942 and the second item tree 944.

[0095] It should be understood that embodiments of this disclosure can be used to perform further iterative boundary expansion in various ways. For example, alternatively... Figure 9D The project trees shown in the image are reset to, respectively, as follows: Figure 9E The state shown at the iteration in step 4 can be... Figure 9D The project tree shown is reset to the state at the iteration of any other predetermined previous step, for example, to the state at the iteration of step 3. Then, the reset project tree can be expanded in the opposite direction to the iteration of the next step following that predetermined previous step. Furthermore, it should be understood that embodiments of this disclosure can also be performed by performing further iteration boundary expansion in various different ways to select the method that yields the best project tree.

[0096] Figure 10 An exemplary boundary detection result according to an embodiment is shown. It is assumed that... Figure 2 Boundary detection was performed on target webpage 20. For example... Figure 10 As shown, dashed box 1010 indicates the item corresponding to "M Phone A4" identified through boundary detection, dashed box 1020 indicates the item corresponding to "M Phone A3" identified through boundary detection, and dashed box 1030 indicates the item corresponding to "M Phone A2" identified through boundary detection. The items indicated by dashed boxes 1010, 1020, and 1030 together form the following... Figure 2 The original list 202 in the target webpage 20 shown.

[0097] Figure 11 An exemplary process 1100, determined according to a master list based on an embodiment, is shown. Process 1100 is... Figure 4 An exemplary implementation of step 430 is described below. Assuming that the target webpage has been determined to include more than one original list, such as a first original list 1102, a second original list 1104, etc., process 1100 can be executed to determine a master list from these original lists.

[0098] At 1110, the visual features of the first original list 1102 can be determined using at least the boundaries of the items in the first original list 1102. At 1120, the visual features of the second original list 1104 can be determined using at least the boundaries of the items in the second original list 1104.

[0099] In embodiments of this disclosure, the visual features of an original list can refer to various visual features that help determine whether the original list occupies a prominent position on a target webpage, whether it is used to present the main content of the target webpage, etc. In one implementation, the visual features may include the minimum boundary distance between adjacent items within the original list, which can indicate the visual distance between the two items. For example, the minimum boundary distance between two adjacent items can be calculated using their boundaries. In one implementation, the visual features may include list position, which can indicate whether the original list occupies a prominent position on the target webpage and thus serves as a major part of the target webpage's content. For example, list position may include the horizontal position of the original list on the target webpage. Alternatively, list position may include the vertical position of the original list on the target webpage, such as whether the list is above the screen's fold line. In one implementation, the visual features may include item content richness, which indicates the visual richness of the items within the original list. For example, the item content richness of an item may include factors such as the item's size and the number of nodes contained in the item, determined based on the item's boundaries.

[0100] At point 1130, a main list can be determined from the first original list 1102 and the second original list 1104 based on the visual features of the first original list 1102 and the second original list 1104. In one implementation, multiple heuristic rules defined for visual features can be used to determine the main list. For example, for minimum boundary distance, a heuristic rule can be defined regarding whether the visual distance between items in the list is small, based on the consideration that the distance between items in the main list is usually not very far. For example, for list position, a heuristic rule can be defined regarding whether the original list occupies a prominent position on the target webpage, based on the consideration that the main list usually occupies a prominent position on the target webpage. For example, for item content richness, a heuristic rule can be defined regarding whether the original list has a high item content richness, based on the consideration that the main list usually has a high item content richness. According to the above heuristic rules, an original list that better satisfies these heuristic rules can be selected from the first original list 1102 and the second original list 1104 as the main list.

[0101] by Figure 2 Taking target webpage 20 as an example, through execution process 1100, original list 202 can be identified as the main list in original list 202 and original list 204.

[0102] It should be understood that all steps in process 1100 are exemplary, and embodiments of this disclosure will also cover any modifications to process 1100. For example, when there are more than two original lists, the master list can be determined by utilizing at least the visual features of these original lists in a manner similar to process 1100. Furthermore, the various visual features and heuristics given above are exemplary, and embodiments of this disclosure may employ any one or more of these visual features and heuristics, or any other type of visual features and heuristics.

[0103] Figure 12 An exemplary process 1200 for obtaining representative metadata according to an embodiment is illustrated. Process 1200 is... Figure 4 An exemplary implementation of step 440 in the above. Assume that process 1200 is performed to obtain a set of representative metadata for a specific item 1202 in an original list.

[0104] At 1210, a set of leaf nodes in the project tree identified by the boundary of project 1202 can be identified. For example, the leaf nodes in the project tree corresponding to project 1202 can be identified.

[0105] At position 1220, a set of initial metadata corresponding to the identified set of leaf nodes can be extracted. For example, the initial metadata for each leaf node can be extracted. Figure 10 Taking the project marked by the dashed box 1010 as an example, the initial metadata may include, for example, the image in the project, the string "M mobile phone A4, 6.5 inches, 256G, black", the icon of 5 solid stars, the string "25900 comments", the string "5500 RMB", etc.

[0106] At point 1230, a set of tags corresponding to the extracted set of initial metadata can be determined. For example, a corresponding tag is assigned to each set of initial metadata to indicate the specific meaning of that initial metadata. This set of tags can be determined at point 1230 in various ways.

[0107] In one implementation, a sequence of tokens can first be formed using the set of initial metadata. Each token in the token sequence corresponds to one of the initial metadata sets. Then, a feature set for each token in the token sequence can be calculated. This feature set can include various features that help determine the token, such as DOM tree features, XPath features, content features, language features, rendering features, etc. DOM tree features can include, for example, the hierarchy depth of the node corresponding to the token, the tag, the category ID, etc. XPath features can include, for example, the name of the node corresponding to the token, the CSS category, etc. Content features can include, for example, the text vector of the token, whether the first letter is capitalized, etc. Language features can include, for example, the language used by the token, the Word2vec semantic feature vector of the token, etc. Rendering features can include, for example, various features involved in rendering the node corresponding to the token, such as position, length, width, etc. It should be understood that embodiments of this disclosure are not limited to the exemplary features included in the feature set given above, but can cover any other features or any combination of these features. A tag for each term can be generated using a pre-trained tagger model based on multiple feature sets of multiple terms in a term sequence. Exemplarily, the tagger model can be a combination of a discrimination model and a generation model, where the discrimination model can be, for example, a binary or multi-class classification model, and the generation model can be, for example, a sequence-to-sequence (Seq2seq) model. It should be understood that embodiments of this disclosure are not limited to generating tags using the tagger model described above, but tags can also be generated in any other manner.

[0108] Still with Figure 10 Taking the project marked by the dashed box 1010 as an example, through step 1230, an "image" tag can be generated for the image in the project, a "title" tag can be generated for the string "M mobile phone A4, 6.5 inches, 256G, black", a "rating" tag can be generated for the icon of 5 solid stars, a "comment" tag can be generated for the string "25900 comments", and a "price" tag can be generated for the string "5500RMB".

[0109] At position 1240, the generated set of tags can be used to sort the initial metadata. In one implementation, a keyword ranking model can be pre-trained to sort the set of tags as keywords. For example, the keyword ranking model can be trained to sort multiple tags according to factors such as importance and representativeness. As an example, for image tags, title tags, rating tags, review tags, price tags, etc., the sorting at position 1240 can sort these tags from high to low as, for example, image tags, title tags, price tags, rating tags, review tags, etc. Accordingly, the initial metadata corresponding to these tags is also sorted in the same order.

[0110] At 1250, one or more of the highest-ranked initial metadata can be selected as a representative set of metadata corresponding to item 1202.

[0111] By executing process 1200 for each item in the original list, multiple sets of representative metadata corresponding to each item in the original list can be obtained. These multiple sets of representative metadata can then be used to generate a structured list.

[0112] It should be understood that all steps in process 1200 are exemplary, and embodiments of this disclosure will also cover any modifications to process 1200.

[0113] According to embodiments of this disclosure, after obtaining multiple sets of representative metadata corresponding to multiple items in the original list, the multiple sets of representative metadata can be visualized as a structured list. Each set of representative metadata can form a new item in the structured list. It should be understood that embodiments of this disclosure are not limited to any particular way of visualizing multiple sets of representative metadata as a structured list. In one implementation, the format or layout of the structured list can be predefined to specify, for example, the arrangement of multiple items in the structured list (e.g., horizontal, vertical, etc.), the arrangement of multiple elements in each item, the size of items and elements, etc. In one implementation, the format or layout of the structured list can be similar to the structure and layout of the original list, except that the structured list may include fewer items or elements than the original list.

[0114] Figure 13 An exemplary search results page 1300 according to an embodiment is shown. Assume a user has entered the query "M mobile phone" in search box 1310, indicating their desire for web search results related to M mobile phones. Search results area 1320 in search results page 1300 includes multiple web search results. (The last sentence appears to be incomplete and possibly refers to a different page.) Figure 3 The region shown in area 330 is for Figure 2 The search results for page 20 are different. Figure 13 Search results area 1320 includes [targets / targets] Figure 2 The exemplary structured list 1330 generated by webpage 20 in the target webpage 20. Structured list 1330 is a simplified version of the original list 202 in the target webpage 20, and can serve as a list summary of the original list 202. Structured list 1330 still contains enough information to allow the user to intuitively and comprehensively understand the main content of the original list 202. For example, structured list 1330 includes items 1332, 1334, and 1336, which correspond to items 1010, 1020, and 1030 respectively in the original list in the target webpage 20 (e.g., ...). Figure 10 (As shown) and includes the main representative content of the corresponding items in the original list. For example, item 1332 includes... Figure 2 The image, description, and price of the "M Phone A4" are presented in area 22. Therefore, users can intuitively and conveniently understand the main content of the target webpage 20 by viewing the structured list 1330 in search results area 1320, without needing to click on a link to the target webpage 20 to access its content. It should be understood that... Figure 13 The search results page 1300 and the structured list 1330 therein are merely exemplary, and the embodiments disclosed herein are not limited to this example.

[0115] Figure 14 A flowchart of an exemplary method 1400 for list extraction and visualization in a webpage, according to an embodiment, is shown.

[0116] At 1410, at least one anchor element group in the target webpage can be detected, the at least one anchor element group including a first anchor element group.

[0117] At 1420, boundary detection can be performed on multiple anchor elements in the first anchor element group to obtain the boundaries of multiple items associated with the multiple anchor elements, the multiple items corresponding to the first original list in the target webpage.

[0118] At point 1430, the boundaries of the multiple items can be used to obtain multiple sets of representative metadata corresponding to the multiple items from the target webpage.

[0119] At 1440, the multiple sets of representative metadata can be visualized as a structured list.

[0120] In one implementation, detecting at least one anchor element group may include: identifying multiple HTML elements in the target webpage that satisfy anchor element constraints as multiple identified anchor elements; extracting the attribute set of each identified anchor element from the target webpage; and clustering the multiple identified anchor elements into the at least one anchor element group based on the multiple attribute sets of the multiple identified anchor elements.

[0121] The anchor element constraints may include at least one of the following: the HTML element has an image tag; the HTML element has a title tag; and the HTML element represents a date. The attribute set for each identifying anchor element may include at least one of the following: the HTML tag attributes, CSS category, and XPath information of the identifying anchor element.

[0122] In one implementation, the boundary detection may include: synchronously performing iterative boundary expansion based on the DOM tree corresponding to the target webpage, starting from the plurality of anchor elements respectively, to obtain multiple item trees respectively originating from the plurality of anchor elements, wherein each item tree represents an item and includes multiple nodes, and each node corresponds to an element determined by the iterative boundary expansion.

[0123] The iterative boundary expansion may include: for each project tree, in each iteration step, expanding to the next node and including the next node in the project tree.

[0124] The iterative boundary expansion may include at least one of the following: performing sibling node expansion to expand from the current node to the sibling nodes of the current node; and performing parent node expansion to expand to the parent node of the current node after all the sibling nodes of the current node have been included in the item tree.

[0125] The boundary detection may include: determining whether the iteration of the current step results in node overlap between the project tree and at least one other project tree among the plurality of project trees; and in response to determining that the node overlap has occurred, stopping the iterative boundary expansion and excluding nodes determined by the iteration of the current step from the plurality of project trees respectively.

[0126] The method may further include: if the iteration of the current step is a sibling node expansion, then performing a further iterative boundary expansion on the multiple project trees in the direction opposite to the direction of the iteration of the current step.

[0127] In one implementation, the boundary detection may include performing a similarity check on the multiple item trees.

[0128] The similarity check may be performed in response to determining that the number of nodes in at least one of the multiple project trees exceeds a node count threshold.

[0129] The similarity check may include: calculating the tree similarity between any two project trees in the plurality of project trees; dividing the plurality of project trees into at least one tree set using at least a similarity threshold, wherein the project trees in each of the at least one tree set have a tree similarity to each other that is not less than the similarity threshold; determining whether the number of project trees in the tree set containing the most project trees in the at least one tree set is less than a tree number threshold; and stopping the iterative boundary expansion in response to determining that the number of project trees is less than the tree number threshold, and excluding nodes determined by a predetermined number of iterations in previous steps from the plurality of project trees respectively.

[0130] The computational tree similarity may include at least one of the following: calculating the tree similarity using at least a matching weight calculated based on the CSS similarity between the root nodes of the two item trees; and calculating the tree similarity using nodes in the two item trees within a minimum depth level, the minimum depth level being defined such that the number of visible nodes within the minimum depth level of an item tree reaches a predetermined proportion of the total number of visible nodes in that item tree.

[0131] The method may further include: resetting the multiple project trees to the state at the iteration of a predetermined previous step; and if the iteration of the next step after the iteration of the predetermined previous step is a sibling node expansion, performing a further iteration boundary expansion on the multiple project trees in the direction opposite to the direction of the iteration of the next step.

[0132] In one implementation, obtaining multiple sets of representative metadata may include, for each of the multiple projects: identifying a set of leaf nodes in a project tree identified by the boundaries of the project; extracting a set of initial metadata corresponding to the set of leaf nodes; determining a set of labels corresponding to the set of initial metadata; sorting the set of initial metadata using the set of labels; and selecting one or more of the highest-ranked initial metadata as a set of representative metadata corresponding to the project.

[0133] The process of determining a set of labels may include: forming a word sequence using the set of initial metadata, where each word in the word sequence corresponds to one of the initial metadata in the set of initial metadata; calculating a feature set for each word in the word sequence; and generating a label for each word based on multiple feature sets of multiple words in the word sequence using a pre-trained labeler model.

[0134] In one implementation, the at least one anchor element group may include a second anchor element group. The method may further include: performing boundary detection on a plurality of anchor elements in the second anchor element group to obtain the boundaries of a plurality of items respectively associated with the plurality of anchor elements, the plurality of items corresponding to a second original list in the target webpage. The method may further include, before obtaining the plurality of sets of representative metadata, or before visualizing the plurality of sets of representative metadata as a structured list: using the boundaries of items in the first original list and the boundaries of items in the second original list, respectively determining the visual features of the first original list and the visual features of the second original list; and based on the visual features of the first original list and the visual features of the second original list, determining that the first original list is the main list in both the first original list and the second original list.

[0135] The visual features may include at least one of the following: minimum boundary distance between adjacent items; list position; and richness of item content.

[0136] In one implementation, the structured list can be presented on a search results page provided by a search service.

[0137] It should be understood that method 1400 may also include any steps / processes for list extraction and visualization in a webpage according to the embodiments of the present disclosure described above.

[0138] Figure 15 An exemplary apparatus 1500 for list extraction and visualization in a webpage is shown according to an embodiment.

[0139] The apparatus 1500 may include: an anchor element group detection module 1510, configured to detect at least one anchor element group in a target webpage, the at least one anchor element group including a first anchor element group; a boundary detection module 1520, configured to perform boundary detection on multiple anchor elements in the first anchor element group to obtain the boundaries of multiple items respectively associated with the multiple anchor elements, the multiple items corresponding to a first original list in the target webpage; a representative metadata acquisition module 1530, configured to obtain multiple sets of representative metadata corresponding to the multiple items from the target webpage using the boundaries of the multiple items; and a representative metadata visualization module 1540, configured to visualize the multiple sets of representative metadata as a structured list.

[0140] In addition, the apparatus 1500 may also include any other modules configured to perform any operations of the methods for list extraction and visualization in web pages according to the embodiments of the present disclosure described above.

[0141] Figure 16 An exemplary apparatus 1600 for list extraction and visualization in a webpage is shown according to an embodiment.

[0142] The apparatus 1600 may include at least one processor 1610. The apparatus 1600 may also include a memory 1620 connected to the at least one processor 1610. The memory 1620 may store computer-executable instructions that, when executed, cause the at least one processor 1610 to: detect at least one group of anchor elements in a target webpage, the at least one group of anchor elements including a first group of anchor elements; perform boundary detection on a plurality of anchor elements in the first group of anchor elements to obtain boundaries of a plurality of items respectively associated with the plurality of anchor elements, the plurality of items corresponding to a first original list in the target webpage; use the boundaries of the plurality of items to obtain a plurality of sets of representative metadata corresponding to the plurality of items respectively from the target webpage; and visualize the plurality of sets of representative metadata as a structured list. Furthermore, the at least one processor 1610 may also be configured to perform any other operations according to the method for list extraction and visualization in a webpage according to the embodiments of the present disclosure described above.

[0143] Embodiments of this disclosure provide a computer program product for list extraction and visualization in web pages. The computer program product includes a computer program executed by at least one processor to: detect at least one group of anchor elements in a target web page, the at least one group of anchor elements including a first group of anchor elements; perform boundary detection on a plurality of anchor elements in the first group of anchor elements to obtain boundaries of a plurality of items respectively associated with the plurality of anchor elements, the plurality of items corresponding to a first original list in the target web page; obtain from the target web page a plurality of sets of representative metadata corresponding to the plurality of items respectively using the boundaries of the plurality of items; and visualize the plurality of sets of representative metadata as a structured list. Furthermore, the computer program may also be executed by the at least one processor to perform any other operations according to the method for list extraction and visualization in web pages according to the embodiments of this disclosure described above.

[0144] Embodiments of this disclosure can be implemented in a non-transitory computer-readable medium. This non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any step / process of the method for list extraction and visualization in a web page according to the embodiments of this disclosure described above.

[0145] It should be understood that all operations in the methods described above are merely exemplary, and this disclosure is not limited to any operation in the methods or the order of such operations, but should cover all other equivalent transformations under the same or similar concept.

[0146] Furthermore, unless otherwise specified or clearly indicated from the context that the singular form is referred to, the articles "a" and "an" as used in this specification and the appended claims should generally be interpreted as meaning "a" or "one or more".

[0147] It should also be understood that all modules in the apparatus described above can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0148] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. As an example, the processors, any portions of processors, or any combinations of processors given in this disclosure can be implemented as microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gate logic, discrete hardware circuits, and other suitable processing units configured to perform the various functions described in this disclosure. The functionality of the processors, any portions of processors, or any combinations of processors given in this disclosure can be implemented as software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0149] Software should be broadly considered as representing instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, procedures, functions, etc. Software may reside on a computer-readable medium. Computer-readable media may include, for example, memory, which may be, for example, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks, smart cards, flash memory devices, random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, or removable disks. Although memory is shown as separate from the processor in several aspects set forth in this disclosure, memory may also reside within the processor (e.g., in caches or registers).

[0150] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or about to be known to those skilled in the art shall be covered by the claims.< / h2>

Claims

1. A method for list extraction and visualization in a webpage, comprising: Detect at least one anchor element group in the target webpage, wherein the at least one anchor element group includes a first anchor element group; Boundary detection is performed on multiple anchor elements in the first anchor element group to obtain the boundaries of multiple items associated with the multiple anchor elements, the multiple items corresponding to the first original list in the target webpage; By utilizing the boundaries of the multiple projects, multiple sets of representative metadata corresponding to each of the multiple projects are obtained from the target webpage; as well as The aforementioned representative metadata sets are visualized as a structured list. The boundary detection includes iterative boundary expansion, and The iterative boundary expansion is performed for each anchor element to find elements within the same item as that anchor element, and the boundary of that item is defined by the anchor element and the found elements.

2. The method as described in claim 1, wherein, The group of at least one anchor element in the target webpage includes: Multiple HTML elements in the target webpage that satisfy the anchor element constraint are identified as multiple anchor elements. Extract the attribute set of each of the plurality of anchor elements from the target webpage; and Based on the multiple attribute sets of the multiple identification anchor elements, the multiple identification anchor elements are clustered into at least one anchor element group.

3. The method of claim 2, wherein, The anchor element constraints include at least one of the following: the html element has an image tag; the html element has a title tag; and the html element represents a date. Each identified anchor element's attribute set includes at least one of the following: the anchor element's HTML tag attributes, CSS category, and XPath information.

4. The method of claim 1, wherein, The boundary detection includes: Based on the Document Object Model (DOM) tree corresponding to the target webpage, the iterative boundary expansion is performed synchronously starting from the plurality of anchor elements to obtain multiple item trees originating from the plurality of anchor elements, wherein each item tree represents an item and includes multiple nodes, and each node corresponds to an element determined by the iterative boundary expansion.

5. The method of claim 4, wherein, The iterative boundary expansion includes: For each project tree, in each iteration of the step, the process expands to the next node and includes that next node in the project tree.

6. The method of claim 5, wherein, The iterative boundary expansion includes at least one of the following: Perform sibling node expansion to expand from the current node to the current node's sibling nodes; as well as Perform parent node expansion to extend the parent node of the current node after all sibling nodes of the current node have been included in the project tree.

7. The method of claim 5, wherein, The boundary detection includes: Determine whether the iteration of the current step results in node overlap between the project tree and at least one other project tree among the plurality of project trees; and In response to the determination that the nodes overlap, the iterative boundary expansion is stopped, and the nodes determined by the iteration of the current step are excluded from the multiple project trees respectively.

8. The method of claim 7, further comprising: If the current iteration is a sibling node expansion, then further iterative boundary expansion is performed on the multiple project trees in the opposite direction to the current iteration.

9. The method of claim 4, wherein, The boundary detection includes: Perform a similarity check on the multiple project trees.

10. The method of claim 9, wherein, The similarity check is performed in response to determining that the number of nodes in at least one of the multiple project trees exceeds a node count threshold.

11. The method of claim 9, wherein, The similarity test includes: Calculate the tree similarity between any two project trees in the plurality of project trees; The multiple item trees are divided into at least one tree set using at least a similarity threshold, wherein the item trees in each tree set have a tree similarity to each other that is not lower than the similarity threshold. Determine whether the number of item trees in the tree set containing the maximum number of item trees in the at least one tree set is less than a tree count threshold; and In response to determining that the number of project trees is lower than the number of trees threshold, the iterative boundary expansion is stopped, and nodes determined by a predetermined number of previous iterations are excluded from the multiple project trees.

12. The method of claim 11, wherein, The calculation of tree similarity between any two project trees in the plurality of project trees includes at least one of the following: The tree similarity is calculated using at least the matching weights based on the CSS similarity between the root nodes of the two project trees; as well as The tree similarity is calculated using nodes within a minimum depth level in the two project trees, where the minimum depth level is defined as such that the number of visible nodes within the minimum depth level of one project tree reaches a predetermined proportion of the total number of visible nodes in that project tree.

13. The method of claim 11, further comprising: Reset the multiple project trees to their state at the iteration of the predetermined previous step; as well as If the iteration of the next step after the iteration of the predetermined previous step is a sibling node expansion, then a further iterative boundary expansion is performed on the multiple project trees in the direction opposite to the direction of the iteration of the next step.

14. The method of claim 1, wherein, The step of obtaining multiple sets of representative metadata corresponding to each of the multiple projects from the target webpage using the boundaries of the multiple projects includes, for each of the multiple projects: Identify a set of leaf nodes in the project tree, as defined by the boundaries of the project; Extract a set of initial metadata corresponding to the set of leaf nodes; Determine a set of tags corresponding to the initial set of metadata; The initial set of metadata is sorted using the set of tags; as well as Select one or more of the highest-ranked initial metadata as a representative set of metadata corresponding to the project.

15. The method of claim 14, wherein, The determination of a set of tags corresponding to the set of initial metadata includes: A term sequence is formed using the set of initial metadata, where each term in the term sequence corresponds to one of the initial metadata items in the set of initial metadata. Calculate the feature set for each term in the term sequence; and A label for each word is generated using a pre-trained labeler model based on multiple feature sets of multiple words in the word sequence.

16. The method of claim 1, wherein, The at least one anchor element group includes a second anchor element group. The method further includes: performing boundary detection on multiple anchor elements in the second anchor element group to obtain the boundaries of multiple items respectively associated with the multiple anchor elements, the multiple items corresponding to the second original list in the target webpage, and The method further includes, before obtaining multiple sets of representative metadata, or before visualizing the multiple sets of representative metadata as a structured list: using the boundaries of items in the first original list and the boundaries of items in the second original list to determine the visual features of the first original list and the visual features of the second original list, respectively; and based on the visual features of the first original list and the visual features of the second original list, determining that the first original list is the main list in the first original list and the second original list.

17. The method of claim 16, wherein, The visual features include at least one of the following: minimum boundary distance between adjacent items; list position; and richness of item content.

18. The method of claim 1, wherein, The structured list is presented on the search results page provided by the search service.

19. An apparatus for extracting and visualizing lists in a webpage, comprising: At least one processor; as well as A memory storing computer-executable instructions that, when executed, cause the at least one processor to: Detect at least one anchor element group in the target webpage, wherein the at least one anchor element group includes a first anchor element group. Boundary detection is performed on multiple anchor elements in the first anchor element group to obtain the boundaries of multiple items associated with each of the multiple anchor elements, the multiple items corresponding to the first original list in the target webpage. By utilizing the boundaries of the multiple projects, multiple sets of representative metadata corresponding to each of the multiple projects are obtained from the target webpage, and The aforementioned representative metadata sets are visualized as a structured list. The boundary detection includes iterative boundary expansion, and The iterative boundary expansion is performed for each anchor element to find elements within the same item as that anchor element, and the boundary of that item is defined by the anchor element and the found elements.

20. A computer program product for list extraction and visualization in a webpage, comprising a computer program that is run by at least one processor for: Detect at least one anchor element group in the target webpage, wherein the at least one anchor element group includes a first anchor element group; Boundary detection is performed on multiple anchor elements in the first anchor element group to obtain the boundaries of multiple items associated with the multiple anchor elements, the multiple items corresponding to the first original list in the target webpage; By utilizing the boundaries of the multiple projects, multiple sets of representative metadata corresponding to each of the multiple projects are obtained from the target webpage; as well as The aforementioned representative metadata sets are visualized as a structured list. The boundary detection includes iterative boundary expansion, and The iterative boundary expansion is performed for each anchor element to find elements within the same item as that anchor element, and the boundary of that item is defined by the anchor element and the found elements.

Images