A catalog information extraction method, device, equipment and medium

Abstract

The application discloses a catalog information extraction method, device and equipment and a medium, and relates to the technical field of information extraction. The method comprises the following steps: obtaining document object model (DOM) information of a target website, determining a structure identifier and corresponding element content from the DOM information, signing the structure identifier and the element content respectively, forming groups of signatures containing the signatures of the structure identifier and the element content, counting the number of pages where the groups of signatures appear and the total number of pages of the website, calculating the coverage of the groups of signatures according to the counted number of pages, and screening out first-level and second-level catalogs according to different coverage intervals. In addition, the term frequency-inverse document frequency (TF-IDF) value of each group of signatures can be obtained, the weight of each group of signatures can be calculated by combining the number of pages where the signatures appear, the catalog can be updated according to the weight order, the similarity between each group of signatures can be calculated, and the signatures with a similarity higher than a threshold value can be merged. The method does not need to customize rules, has strong adaptability, can generate a catalog with clear levels and conforming to the logic of the website, reduces the maintenance cost, and improves the extraction accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information extraction, and in particular to a catalog information extraction method and device, equipment and a medium. BACKGROUND

[0002] With the development of the Internet, platforms such as enterprise websites often classify products according to different types and present them in the form of a catalog in order to facilitate user search and browsing. Product type catalog information (such as product categories, subcategories, and corresponding hierarchical relationships) is a core element for users to quickly locate target products and achieve efficient navigation. It is also important basic data for enterprises to manage products, segment markets, and analyze competitors. Therefore, accurate and efficient extraction of such information is of great significance.

[0003] With the rapid increase in the number of websites and the continuous enrichment of product types, the structure of product type catalogs is becoming increasingly complex and diverse. How to accurately extract product type catalog information from websites with different architectures has become a problem that needs to be solved in the field of information extraction. SUMMARY

[0004] The present application provides a catalog information extraction method, device, equipment and medium, which can accurately extract product type catalog information from websites with different architectures.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a catalog information extraction method, comprising:

[0007] Obtaining the Document Object Model (DOM) information of a target website;

[0008] Determining structural identifiers from the DOM information and determining the element content corresponding to the structural identifiers from the DOM information;

[0009] Signing each structural identifier to obtain the structural signature corresponding to each structural identifier, and signing each element content to obtain the content signature corresponding to each element content;

[0010] Obtaining the number of pages where each group of signatures appears and the total number of pages of the target website, each group of signatures consisting of the structural signature corresponding to a structural identifier and the content signature of the element content corresponding to the structural identifier;

[0011] Determining the coverage of each group of signatures according to the number of pages where each group of signatures appears and the total number of pages of the target website;

[0012] Obtaining the catalog information of the target website according to the coverage of each group of signatures.

[0013] Optionally, the method further comprises:

[0014] obtaining a term frequency-inverse document frequency (TF-IDF) value of each group of signatures;

[0015] determining a weight of each group of signatures according to the TF-IDF value of each group of signatures and a number of pages where each group of signatures appears;

[0016] updating the directory information of the target website according to a size of the weight of each group of signatures.

[0017] Optionally, the method further comprises:

[0018] calculating a similarity between each two groups of signatures;

[0019] merging two groups of signatures with a similarity greater than a similarity threshold into one group of signatures.

[0020] Optionally, the weight of each group of signatures is determined according to the TF-IDF value of each group of signatures and the number of pages where each group of signatures appears, and the method comprises:

[0021]

[0022] wherein, denotes the weight of the nth group of signatures, denotes a set of pages where the nth group of signatures appears, denotes the number of pages where the nth group of signatures appears, denotes a set of words in the nth group of signatures, denotes the number of words in the nth group of signatures, denotes a TF-IDF value of the tth word in the nth group of signatures.

[0023] Optionally, the coverage of each group of signatures is determined according to the number of pages where each group of signatures appears and a total number of pages of the target website, and the method comprises:

[0024]

[0025] wherein, denotes the coverage of the nth group of signatures, denotes the number of pages where the nth group of signatures appears, denotes the total number of pages of the target website.

[0026] Optionally, the directory information of the target website is obtained according to the coverage of each group of signatures, and the method comprises:

[0027] Filtering a first level signature with a coverage rate in a first coverage rate interval from the groups of signatures, taking the first level signature as a first level directory of the target website, filtering a second level signature with a coverage rate in a second coverage rate interval from the groups of signatures, taking the second level signature as a second level directory of the target website, the first level directory being higher in level than the second level directory, the minimum value of the first coverage rate interval being greater than the maximum value of the second coverage rate interval.

[0028] Optionally, the determining the structural identifier from the dom information comprises:

[0029] Obtaining a start tag and an end tag corresponding to the start tag in the dom information;

[0030] Removing a preset character in the end tag to obtain the structural identifier.

[0031] In a second aspect, the present application provides a directory information extraction device, comprising:

[0032] An obtaining module, configured to obtain a document object model (dom) information of a target website, and obtain a page number of each group of signatures and a total page number of the target website, each group of signatures being composed of a structural signature corresponding to a structural identifier and an element signature corresponding to element content of the structural identifier;

[0033] A processing module, configured to determine a structural identifier from the dom information, and determine element content corresponding to the structural identifier from the dom information, sign each structural identifier to obtain a structural signature corresponding to each structural identifier, sign each element content to obtain an element signature corresponding to each element content, and determine a coverage rate of each group of signatures according to the page number of each group of signatures and the total page number of the target website;

[0034] An output module, configured to obtain directory information of the target website according to the coverage rate of each group of signatures.

[0035] In a third aspect, the present application provides a computing device, comprising a memory and a processor;

[0036] Wherein, the memory stores one or more computer programs, the one or more computer programs comprising instructions; when the instructions are executed by the processor, the computing device executes the method of any one of the first aspect.

[0037] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium is used to store a computer program, the computer program is used to execute the method of any one of the first aspect.

[0038] From the above technical solution, the present application has at least the following beneficial effects:

[0039] In the present application, first, the structural identifier and element content are obtained by dom information analysis, and then the abstract directory structure is converted into quantifiable and comparable feature information through the dual identification of structure signature and content signature, effectively avoiding the extraction deviation caused by the structural differences in different website dom information; at the same time, combined with coverage calculation (based on the ratio of the number of pages with signature to the total number of pages of the website), the directory information with universal representation in the website can be screened out, avoiding the interference of non-core and scattered information.

[0040] Further, by dividing the first-level directory and the second-level directory according to the coverage interval, a logical and clear directory hierarchy relationship can be automatically constructed, without manual manual sorting, greatly reducing the labor cost; the similarity merging step can merge the signatures with similar content or structure, avoiding the repetition and redundancy of directory information, and further improving the directory quality; in addition, the TF-IDF value is introduced to calculate the signature weight, which can preferentially retain the directory information with high semantic value and close association with the product, so that the extracted directory information is more accurate.

[0041] The method does not need to rely on the fixed architecture or preset rules of a specific website, and through the general logic of dom information analysis and signature analysis, it can adapt to websites with different architectures and different product types, solving the extraction problem caused by the rapid increase in the number of websites and the diversity of directory structures; the extracted directory information can not only meet the navigation needs of users to quickly locate target products, but also provide accurate and structured basic data for enterprise product management, market segmentation and competitor analysis.

[0042] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in the present application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of a feature or a beneficial effect means that the specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in the specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and beneficial effects described in the present embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of a directory information extraction method provided by the present embodiment;

[0044] Figure 2 FIG. 1 is a schematic diagram of a directory information extraction device according to an embodiment of the present application;

[0045] Figure 3 FIG. 2 is a schematic diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The terms “first”, “second”, and “third” and the like in the specification and the drawings of the present application are used to distinguish different objects, and are not used to limit a specific order.

[0047] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration, in no way limiting. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design schemes. Rather, the use of the words “exemplary” or “for example” is intended to present relevant concepts in a concrete manner.

[0048] For the sake of clear and concise description of the following embodiments, first, a brief introduction of related technologies is given:

[0049] The document object model (DOM) information refers to converting web page content into a structured tree-shaped data model. Each element in the web page is a node in the tree, containing information such as tags, attributes, and text content, and is the core basis for computer analysis of web page structure, equivalent to the skeleton blueprint of the web page.

[0050] However, in the current practice of extracting website product type directory information, two mainstream traditional methods essentially need to be based on DOM information for analysis and processing. One is the rule-based extraction method, in which technical personnel analyze the HTML code structure of the website. This structure is actually a direct representation of DOM information, and clearly indicates the label position, hierarchical nesting relationship, etc. of the directory information in the web page, and then corresponding extraction rules (such as specific label matching rules, hierarchical path rules, etc.) are written to extract product type directory information. The other is to use a large model for extraction, which takes advantage of the large model's understanding of natural language to analyze the text content related to product types in the web page, and thus identify the product type directory and its hierarchical relationship.

[0051] But the two traditional extraction methods have defects that cannot be ignored. The rule-based extraction method is severely limited by the structure of the website. Once the website adjusts the display style of the directory, the layout of the label, etc., the original extraction rule will lose its effect, and the rule needs to be rewritten. Moreover, this method is only suitable for product type directories with structured and clear levels. For directories with chaotic structure, ambiguous levels, or using dynamic loading technology, the extraction effect is poor. Although the large model extraction method can cope with a certain degree of structural changes, it often deviates in the level division and category definition of the product type directory. Due to the differences in naming habits and classification logic of product types on different websites, the large model is prone to misjudgment of categories and confusion of level relationships when processing these personalized information, resulting in inaccurate extracted directory information.

[0052] Therefore, the embodiments of the present application provide an extraction method of directory information, which can be executed by a processing device. The processing device can be a terminal or a server. The terminal includes but is not limited to a smartphone, a tablet computer, a notebook computer, a personal digital assistant, or a smart wearable device, etc. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Of course, the server can also be a server in a local data center. The local data center refers to a data center directly controlled by the user.

[0053] The embodiments of the present application provide an extraction method of directory information. First, the structured characteristics of the webpage dom information are used to extract the structural identifiers and element contents that can represent the directory carrier, breaking through the limitations of different website architectures and realizing structural adaptation. Second, the directory-related information is converted into quantifiable features for comparison through dual encoding of structural signatures and content signatures, providing a basis for subsequent screening and merging. Third, the coverage, TF-IDF value, and signature weight are introduced to establish a multi-dimensional evaluation method to accurately distinguish core directories from non-core information. Finally, the directory hierarchy is divided by coverage interval, and repeated information is merged by similarity calculation to realize automatic optimization of the directory structure, and then the directory information of the target website is obtained.

[0054] After the basic extraction and optimization process, the final directory information of the target website is obtained, which contains clear directory levels (one level, two levels, or multiple levels) and non-repeated core directory content, and can be directly used for user product navigation or enterprise product management, market analysis, etc.

[0055] In order to make the technical solutions of the present application clearer and easier to understand, the following will introduce the extraction method of directory information provided by the embodiments of the present application in conjunction with the drawings. As shown in the figure, it is a flowchart of the extraction method of directory information provided by the embodiments of the present application. The method includes: Figure 1 ​

[0056] S201, the processing device obtains the document object model (DOM) information of the target website.

[0057] The target website is an enterprise website that needs to extract product classification information, and is the source carrier of DOM information. The HTML structure of the target website is heterogeneous. For example, some websites use nested tables to build navigation, and some use tags to carry classification. The target website includes home page, classification page, product detail page and other full-level pages.

[0058] The processing device takes the home page URL of the target website as the entry, first executes the depth-first traversal strategy, and crawls the full path of the home page, classification list page and product detail page to ensure that all levels of pages of the website are covered and product classification related pages are not missed. The cluster parallel crawling capability is used to simultaneously process the requests and data returns of multiple web pages. For large-scale enterprise websites with more than 1000 pages, the collection speed can be increased by 5-10 times. Finally, the original HTML set of the target website is obtained, which contains the complete HTML code of all pages.

[0059] Then, the original HTML obtained by crawling is preprocessed to remove irrelevant tags and their subtrees, including script, style, svg, iframe, advertisement container, footer, img and other non-content related elements. These elements do not contain product classification information and will increase the noise and computational complexity of subsequent processing. The HTML structure strongly related to product classification is retained, for example <header>top navigation bar, breadcrumb navigation tags, <sidebar>side classification list, / Product classification lists, etc., convert the processed HTML into a structured DOM information tree model.

[0060] Finally, a purified DOM information set is obtained. This set removes approximately 70% of the irrelevant content and only retains the DOM information structure in the core content area, which not only reduces the computational complexity of subsequent discovery of high-frequency directory segments but also improves the accuracy of directory element recognition.

[0061] S202. The processing device determines a structure identifier from the DOM information and determines the element content corresponding to the structure identifier from the DOM information.

[0062] Specifically, first obtain the start tag in the DOM information and the end tag corresponding to the start tag; then remove the preset characters in the end tag to obtain the structure identifier.

[0063] The start tag and the end tag are paired tags that identify the boundaries of elements in HTML. The start tag is the first "<" that appears in each line. For example, the "<" in "<u……” represents the start tag; the end tag is "”", where "x" is a general term. For example, the end tag can be "”" or.

[0064] In some examples, taking the DOM information <a href=""...”">Content example ” as an example, the definitions of each part in the DOM information are introduced. See the following table:

[0065]

[0066] The preset character is the key symbol that distinguishes the start tag from the end tag. For example in the end tag.

[0067] The structure identifier is obtained by processing the end tag, and the preset character is " / " in the end tag. The result identifier can be obtained by removing " / " in the end tag, for example, the result identifier obtained by removing " / " in the end tag is " and the processed result is obtained ", which can be used to locate elements with the same structure type in the dom information.

[0068] The processing device traverses all element nodes from the dom information tree model, identifies and extracts the start tag and the corresponding end tag of each element, ensures the accurate correspondence of the paired tags, and lays a foundation for subsequent structure identification.

[0069] For the extracted end tag, the " / " in it, i.e. the preset character, is removed by a preset rule to obtain a structure identifier representing the structure type of the element, thereby realizing the unified identification of different dom information structures. Taking the structure identifier as an index, the region wrapped by the corresponding start tag and end tag is located, the pure text information is extracted from the region, and the non-text content such as nested tags is filtered out to obtain the element content. Through this process, the complex structured information in the dom information can be disassembled into the structure identifier and the element content.

[0070] S203, the processing device signs each structure identifier to obtain a structure signature corresponding to each structure identifier, and signs each element content to obtain a content signature corresponding to each element content.

[0071] The processing device signs the whole of each structure identifier and the attribute thereof extracted, for example ”、" "and" ", etc. are calculated by using a preset hash algorithm. The algorithm generates a unique string of fixed length, i.e. the structure signature, according to the features of the structure identifier and the character composition, length, attribute key-value pair, etc. of the attribute. For example, for the attribute "name = "John Doe", the structure signature is calculated as "John Doe". " a3b9c2... " after signing the " whole " against The overall signature can result in "f7e1d5...". With structural signature, it is possible to quickly identify whether there are the same type of structure and attribute combination in different pages, such as two pages of " The structural signatures of the labels are consistent, indicating that they belong to the same category of structures with specific attributes.

[0072] The processing device also calculates a unique content signature for each extracted element content, such as "smart door lock" and "office equipment", using a hash algorithm. For example, the content signature of "smart door lock" might be "d2c3e1..." and the content signature of "office equipment" might be "b5a7f9...". The content signature can be used to quickly determine whether the text content in different pages is the same. For example, if the content signatures of "smart door lock" in two pages are consistent, it means that they are the same directory content.

[0073] By generating structural signatures and content signatures, the processing device converts the originally difficult-to-compare structural identifiers and text content into computable and matchable digital fingerprints.

[0074] Calculate the similarity between each pair of signatures in each group of signatures; merge two signatures with a similarity greater than the similarity threshold into one group of signatures.

[0075] This process is a key step after high-frequency directory fragment discovery and before multi-source tree fusion. The purpose is to eliminate duplicate or highly similar directory signatures and reduce redundant calculations in subsequent tree fusion.

[0076] Calculate the similarity between each pair of signatures in each group of signatures; merge two signatures with a similarity greater than the similarity threshold into one group of signatures.

[0077] Structural similarity calculation: Extract the MinHash structural signature vector (128 dimensions) of the two groups of signatures, and calculate the coincidence degree between the vectors, such as the matching number of the same position minimum hash value, to get the structural similarity. For example, if the MinHash vector of the two groups of signatures has 108 identical position values, then the structural similarity is 108 / 128≈0.84.

[0078] Content similarity calculation: Extract the SimHash content signature (128 dimensions) of the two groups of signatures, and calculate the Hamming distance (number of different positions) and convert it to similarity. For example, if the Hamming distance of the two SimHash signatures is 16, then the content similarity is 1-16 / 128=0.875.

[0079] Comprehensive similarity calculation: Combine the weights of structural and content similarity, such as 50% for structure and 50% for content, to get the overall similarity of the two groups of signatures. If the structural similarity is 0.84 and the content similarity is 0.875, then the comprehensive similarity is (0.84+0.875) / 2≈0.857.

[0080] Set 0.75 as the comprehensive similarity threshold, if the comprehensive similarity of two groups of signatures is higher than 0.75, and the structural similarity is greater than 0.7 and the content similarity is greater than 0.8, it is determined that the signatures are highly similar, and the two groups of highly similar signatures are combined into one group of signatures.

[0081] In S204, the processing device obtains the number of pages where each group of signatures appears and the total number of pages of the target website, each group of signatures being composed of a structural signature corresponding to a structural identifier and a content signature of element content corresponding to the structural identifier.

[0082] The processing device combines the structural signature and the content signature corresponding to the same structural identifier to form a group of signatures. This combination can reflect both the structural characteristics and the content characteristics of the information, ensuring that each group of signatures corresponds to unique directory information.

[0083] The processing device traverses all pages of the target website, identifies and records the group signatures in each page, counts the number of times each group of signatures appears in different pages, and only counts the same group of signatures appearing repeatedly in the same page once to avoid repeated counting. Finally, the number of pages where each group of signatures appears, i.e., the number of pages where the group of signatures exists, is obtained.

[0084] The processing device directly calls the total number of pages of the target website recorded in the pre-crawling stage as the reference data for subsequent calculation.

[0085] In S205, the processing device determines the coverage of each group of signatures based on the number of pages where each group of signatures appears and the total number of pages of the target website.

[0086] Coverage is a core indicator for measuring the popularity of a group of signatures in a target website, reflecting the universal value of the directory information corresponding to the group of signatures in the website. The higher the coverage, the more likely it is that the directory information is a globally universal core classification content of the website, rather than scattered information appearing only in individual pages.

[0087] The calculation expression is:

[0088]

[0089] wherein, represents the coverage of the nth group of signatures, represents the number of pages where the nth group of signatures appears, represents the total number of pages of the target website.

[0090] For highly similar signatures, the merging rule is to preferentially retain the structural and content characteristics of the signature with higher coverage.

[0091] For example, the comprehensive similarity between the smart phone (coverage 45%) and the smart phone (coverage 43%) is 0.82, which exceeds the comprehensive similarity threshold 0.75, and after merging, the unified signature corresponding to the smart phone is formed, covering the page range of the original two groups of signatures.

[0092] In S206, the directory information of the target website is obtained according to the coverage of each group of signatures.

[0093] The directory information refers to the hierarchical information of product classification in the target website, including the major category, the minor category, the sub-category, and their affiliation, which is the core reference data for user navigation and enterprise management.

[0094] The first-level signatures with the coverage in the first coverage interval are screened out from the signatures, and the first-level signatures are taken as the first-level directory of the target website. The second-level signatures with the coverage in the second coverage interval are screened out from the signatures, and the second-level signatures are taken as the second-level directory of the target website. The level of the first-level directory is higher than that of the second-level directory, and the minimum value of the first coverage interval is greater than the maximum value of the second coverage interval.

[0095] The first coverage interval and the second coverage interval are preset coverage value ranges for dividing the directory level, wherein the minimum value of the first coverage interval is greater than the maximum value of the second coverage interval. By the interval threshold difference, the general degree of the directory information is distinguished. The first coverage interval corresponds to the core information that appears globally and frequently in the website, and the second coverage interval corresponds to the subdivided information that is less general than the general range. For example, the first coverage interval is (80%, 100%], and the second coverage interval is [60%, 80%].

[0096] The first-level signatures and the second-level signatures respectively refer to the combination of the structural signatures and the content signatures with the coverage falling into the first coverage interval and the second coverage interval.

[0097] The first-level directory and the second-level directory are two core levels of the directory system. The first-level directory is the top core classification, and the second-level directory is the subdivided classification under the first-level directory. The two are in a subordinate relationship, and the level of the first-level directory is higher than that of the second-level directory.

[0098] If the coverage of a group of signatures is in the interval (80%, 100%], the first-level signatures are screened out. The element content corresponding to these signatures is directly taken as the first-level directory of the target website. These directories are the top-level classifications that are globally common in the website and appear in all core pages (such as the home page and the classification page).

[0099] Similarly, if the coverage of a group of signatures is in the interval [60%, 80%], the second-level signatures are screened out. The element content corresponding to these signatures is taken as the second-level directory. These contents are subdivided classifications and only appear frequently in the related pages corresponding to the first-level directory.

[0100] In addition, if the coverage of a group of signatures is in the interval [40%, 60%), the screening is the third level signature, and the element label is included in other elements not covered by the first level and second level signatures; the element content is extracted as a third level directory candidate. Such content is a more detailed classification under the second level directory, and only appears in the subdivision page corresponding to the second level directory, further refining the product classification dimension.

[0101] The hierarchical relationship of the web page directory is reflected by the nesting of the dom information elements. The element of the first level directory is usually the parent node or ancestor node of the second level directory element in the dom information tree. The processing device records the complete path of the first level and second level directory elements in the dom information. If the path of the second level directory element belongs to the sub-path of the path of the first level directory element, it is preliminarily determined that there is a possible attribution between them.

[0102] For example, the dom information path corresponding to the electronic product in the first level directory is "html→body→header→div.top-cate", and the dom information path corresponding to the mobile phone in the subordinate second level directory is "html→body→header→div.top-cate→ul→li.sub-cate".

[0103] Among them, the former part "html→body→header→div.top-cate" in "html→body→header→div.top-cate→ul→li.sub-cate" is the dom information path corresponding to the electronic product. As can be seen, the dom information path of the mobile phone is an extended sub-path of the path of the electronic product, which clearly shows that the mobile phone element is nested in the electronic product element, and it can be preliminarily determined that the first level directory to which the mobile phone belongs is the electronic product.

[0104] Through coverage screening, a directory framework with clear hierarchical relationship and clear attribution relationship has been preliminarily constructed. In order to further quantify the core value of the directory information, optimize the priority order of the directory system, and ensure that the final directory not only meets the global logic of the website, but also highlights the high-value classification content, the method further includes:

[0105] The processing device first obtains the term frequency-inverse document frequency TF-IDF value of each group of signatures.

[0106] The TF-IDF value is a comprehensive score of the importance of the signature. The higher the TF-IDF value of a group of signatures, the more frequently it appears in a specific web page, and the stronger the discrimination in the overall web page set, the more likely it is the core directory content corresponding to the web page.

[0107] By obtaining the TF-IDF values, the processing device can filter out core signatures more valuable for directory system construction from a large number of signatures, avoid interference by general or redundant information, and provide a quantitative basis for subsequent determination of directory level priorities and optimization of multi-source tree fusion results.

[0108] According to the TF-IDF values of the signatures in each group and the number of pages in which the signatures in each group appear, the weight of each group of signatures is determined.

[0109] The calculation expression is:

[0110]

[0111] wherein, Wn represents the weight of the n-th group of signatures, Sn represents a set of pages in which the n-th group of signatures appears, Nn represents the number of pages in which the n-th group of signatures appears, Vn represents a set of words in the n-th group of signatures, Mn represents the number of words in the n-th group of signatures, Tn,t represents the TF-IDF value of the t-th word in the n-th group of signatures.

[0112] According to the size of the weight of each group of signatures, the directory information of the target website is updated.

[0113] First, the weight of each group of signatures is sorted from high to low, and the higher the weight, the more general and core the directory information corresponding to the signature.

[0114] If there are signatures with similar weights and similar semantics after sorting, for example, smart phone and smartphone, the weight of smart phone is 3.18, and the weight of smartphone is 3.21, the weight of smart phone is slightly lower, which is determined as redundant information, and only the smartphone with higher weight is retained to avoid directory duplication.

[0115] According to the sorting result, the directory is updated: the high-weight signature is used as a first-level directory, the medium-weight signature corresponds to a second-level directory, and the low-weight signature corresponds to a third-level and below directory; at the same time, the repeated and mismatched directory attribution is removed, and finally a target website directory system with clear levels and prominent cores is formed.

[0116] Based on the above content description, the present application has the following beneficial effects:

[0117] In the present application, first, the structural identifier and element content are obtained through dom information analysis, and then the abstract directory structure is converted into quantifiable and comparable feature information through the double identification of structure signature and content signature, effectively avoiding the extraction deviation caused by the difference of dom information structure of different websites; at the same time, combined with coverage calculation (based on the ratio of the number of pages where the signature appears to the total number of pages of the website), the directory information with universal representation in the website can be accurately screened out, avoiding the interference of non-core and scattered information.

[0118] Further, by dividing the first-level and second-level directories according to the coverage interval (the minimum value of the coverage interval of the first-level directory is greater than the maximum value of the second-level directory), a logical and clear directory hierarchy can be automatically constructed, without manual manual sorting, greatly reducing the labor cost; the similarity merging step can merge the signatures with similar content or structure, avoiding the repetition and redundancy of directory information, and further improving the directory quality; in addition, the TF-IDF value is introduced to calculate the signature weight, which can preferentially retain the directory information with high semantic value and close association with the product, so that the finally extracted directory is more in line with the user navigation and enterprise management needs.

[0119] The method does not need to rely on the fixed architecture or preset rules of a specific website, and through the general logic of dom information analysis and signature analysis, it can adapt to websites with different architectures and different product types, solving the extraction problem caused by the rapid increase in the number of websites and the diversity of directory structure; the finally extracted directory information can not only meet the navigation needs of users to quickly locate target products, but also provide accurate and structured basic data for enterprise product management, market segmentation and competitor analysis, realizing the dual landing of technical value and business value.

[0120] The above is combined Figure 1 The directory information extraction method provided by the embodiments of the present application is described in detail, and the device and equipment provided by the embodiments of the present application will be introduced below with reference to the drawings.

[0121] As Figure 2 shown, the figure is a schematic diagram of a directory information extraction device provided by an embodiment of the present application, which comprises:

[0122] The acquisition module 301 is configured to acquire the document object model (dom) information of a target website, and acquire the number of pages where each group of signatures appears and the total number of pages of the target website, each group of signatures being composed of a structure signature corresponding to a structural identifier and a content signature of element content corresponding to the structural identifier;

[0123] The processing module 302 is configured to determine a structure identifier from the DOM information, determine element content corresponding to the structure identifier from the DOM information, sign each structure identifier to obtain a structure signature corresponding to each structure identifier, sign each element content to obtain a content signature corresponding to each element content, and determine a coverage rate of each group of signatures according to a number of pages in which each group of signatures appears and a total number of pages of the target website.

[0124] The output module 303 is configured to obtain directory information of the target website according to the coverage rate of each group of signatures.

[0125] Optionally, the acquisition module 301 is further configured to acquire a term frequency-inverse document frequency (TF-IDF) value of each group of signatures.

[0126] The processing module 302 is further configured to determine a weight of each group of signatures according to the TF-IDF value of each group of signatures and the number of pages in which each group of signatures appears, sort the weights of each group of signatures according to the size, and update the directory information of the target website.

[0127] Optionally, the processing module 302 is further configured to calculate a similarity between each two groups of signatures in each group of signatures, and combine two groups of signatures with a similarity greater than a similarity threshold into one group of signatures.

[0128] Optionally, the processing module 302 is specifically configured to determine the weight of each group of signatures according to the TF-IDF value of each group of signatures and the number of pages in which each group of signatures appears, including:

[0129]

[0130] wherein, denotes the weight of the nth group of signatures, denotes a set of pages in which the nth group of signatures appears, denotes the number of pages in which the nth group of signatures appears, denotes a set of words in the nth group of signatures, denotes the number of words in the nth group of signatures, denotes a TF-IDF value of the tth word in the nth group of signatures.

[0131] Optionally, the processing module 302 is specifically configured to determine the coverage rate of each group of signatures according to the number of pages in which each group of signatures appears and the total number of pages of the target website, including:

[0132]

[0133] wherein, denotes the coverage rate of the nth group of signatures, denotes the number of pages in which the nth group of signatures appears, The total page number of the target website is represented.

[0134] Optionally, the processing module 302 is specifically configured to filter first-level signatures with coverage in a first coverage interval from the signatures in each group, take the first-level signatures as first-level directories of the target website, filter second-level signatures with coverage in a second coverage interval from the signatures in each group, and take the second-level signatures as second-level directories of the target website. The first-level directories have a higher level than the second-level directories, and the minimum value of the first coverage interval is greater than the maximum value of the second coverage interval.

[0135] Optionally, the acquisition module 301 is specifically configured to acquire a start tag and an end tag corresponding to the start tag in the dom information.

[0136] The processing module 302 is specifically configured to remove a preset character in the end tag to obtain a structure identifier.

[0137] The directory information extraction apparatus according to the embodiments of the present application can correspond to the method described in the embodiments of the present application, and the above-mentioned other operations and / or functions of each module / unit of the directory information extraction apparatus are respectively implemented to realize the corresponding flow of each method in the embodiments of the present application. For brevity, they will not be repeated here. Figure 1 The above-mentioned other operations and / or functions of each module / unit of the directory information extraction apparatus are respectively implemented to realize the corresponding flow of each method in the embodiments of the present application. For brevity, they will not be repeated here.

[0138] The embodiments of the present application also provide a computing device. As shown in the Figure 3 The figure is a schematic diagram of a computing device provided by the embodiments of the present application. The computing device 700 includes a bus 701, a processor 702, a communication interface 703 and a memory 704. The processor 702, the memory 704 and the communication interface 703 communicate through the bus 701.

[0139] The bus 701 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 3 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0140] The processor 702 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0141] The communication interface 703 is configured to communicate with the outside.

[0142] The memory 704 can include a volatile memory, such as a random access memory (RAM) and / or a non-volatile memory, such as a read-only memory (ROM), a floppy disk, a hard disk, or a solid state drive (SSD).

[0143] The memory 704 stores executable code, and the processor 702 executes the executable code to perform the aforementioned method for extracting directory information.

[0144] Specifically, in the case of implementing the embodiment shown in the Figure 2 In the case of implementing the embodiment shown in the Figure 2 In the case of implementing the embodiment shown in the Figure 2 In the case of implementing the embodiment shown in the

[0145] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can store data which can be accessed by a computing device, such as a data center or the like containing one or more available media. The available media can be a magnetic medium (e.g., a floppy diskette, a hard disk drive), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state hard drive) or the like. The computer readable storage medium includes instructions that instruct a computing device to perform the aforementioned method for extracting directory information.

[0146] The embodiments of the present application further provide a computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated.

[0147] The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line) or wireless (for example, infrared, wireless, microwave, etc.).

[0148] The computer program product is executed by a computer, and the computer executes any of the directory information extraction methods described above. The computer program product can be a software installation package, and when any of the directory information extraction methods described above is needed, the computer program product can be downloaded and executed on the computer.

[0149] The descriptions of the processes or structures corresponding to the above respective figures are each focused on, and the parts not described in detail in a certain process or structure can be referred to the related descriptions of other processes or structures.

[0150] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application. < / sidebar> < / header>

Claims

1. A method for extracting directory information, characterized in that, The method includes: Obtain the Document Object Model (DOM) information of the target website; Determine the structure identifier from the DOM information, and determine the element content corresponding to the structure identifier from the DOM information; Sign each structure identifier to obtain the structure signature corresponding to each structure identifier, and sign each element content to obtain the content signature corresponding to each element content. Obtain the number of pages where each group of signatures appears and the total number of pages on the target website. Each group of signatures consists of a structural signature corresponding to a structural identifier and a content signature of the element content corresponding to the structural identifier. The coverage rate of each signature group is determined based on the number of pages in which each signature appears and the total number of pages on the target website. Based on the coverage of each group of signatures, the directory information of the target website is obtained; Based on the coverage of each signature group, the directory information of the target website is obtained, including: First-level signatures with coverage in the first coverage range are selected from each group of signatures and used as the first-level directory of the target website. Second-level signatures with coverage in the second coverage range are selected from each group of signatures and used as the second-level directory of the target website. The first-level directory has a higher level than the second-level directory. The minimum value of the first coverage range is greater than the maximum value of the second coverage range.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the term frequency-inverse document frequency (TF-IDF) values ​​for each signature group; The weight of each signature group is determined based on the TF-IDF value of each signature group and the number of pages in which each signature group appears. The directory information of the target website is updated by sorting the signatures of each group according to their weight.

3. The method according to claim 1, characterized in that, The method further includes: Calculate the similarity between any two groups of signatures in each group; Two sets of signatures with a similarity greater than the similarity threshold are merged into one set of signatures.

4. The method according to claim 2, characterized in that, The weight of each signature group is determined based on its TF-IDF value and the number of pages it appears on, including: in, This represents the weight of the nth signature group. This represents the set of pages where the nth signature appears. This represents the number of pages where the nth signature appears. This represents the set of words in the nth signature. This represents the number of words in the nth signature group. This represents the TF-IDF value of the t-th word in the n-th tag group.

5. The method according to claim 1, characterized in that, The coverage of each signature group is determined based on the number of pages in which each signature appears and the total number of pages on the target website, including: in, This represents the coverage rate of the nth signature group. This represents the number of pages where the nth signature appears. This indicates the total number of pages on the target website.

6. The method according to claim 1, characterized in that, Determining the structure identifier from the DOM information includes: Obtain the start tag and the corresponding end tag from the DOM information; Remove the preset characters from the closing tag to obtain the structure identifier.

7. A device for extracting directory information, characterized in that, The device includes: The acquisition module is used to acquire the Document Object Model (DOM) information of the target website; acquire the number of pages where each group of signatures appears and the total number of pages of the target website. Each group of signatures consists of a structural signature corresponding to a structural identifier and a content signature of the element content corresponding to the structural identifier. The processing module is used to determine the structure identifier from the DOM information, determine the element content corresponding to the structure identifier from the DOM information; sign each structure identifier to obtain the structure signature corresponding to each structure identifier, sign each element content to obtain the content signature corresponding to each element content; and determine the coverage of each set of signatures based on the number of pages in which each set of signatures appears and the total number of pages of the target website. The output module is used to obtain the directory information of the target website based on the coverage of each group of signatures. Specifically, it is used to filter out the first-level signatures with a coverage rate in the first coverage range from each group of signatures and use the first-level signatures as the first-level directory of the target website. It is also used to filter out the second-level signatures with a coverage rate in the second coverage range from each group of signatures and use the second-level signatures as the second-level directory of the target website. The level of the first-level directory is higher than the level of the second-level directory. The minimum value of the first coverage range is greater than the maximum value of the second coverage range.

8. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 6.

Images