An information extraction method, device, equipment and medium

Abstract

The application discloses an information extraction method and device, equipment and medium, and relates to the technical field of electronic commerce. The scheme can reduce the number of searches, reduce traffic consumption, and improve identification accuracy by automatically collecting target sites, determining the types of the target sites by using a rule-based hierarchical feature detection technology. According to the types of the target sites, the links of the pages in the corresponding sites are obtained, efficient collection of product detail page links is realized, manual intervention of personnel is reduced, and the efficiency and accuracy of the extraction in large-scale sites are greatly improved. Meanwhile, the pages corresponding to all the links are clustered and classified, the automation of page classification and identification is realized, and finally, target information is identified based on the target pages. The scheme greatly improves the data extraction efficiency and quality in large-scale e-commerce sites through automatic and intelligent technical means, can meet the demand for real-time data acquisition, and is particularly suitable for large-scale data extraction.

Description

Technical Field

[0001] This application relates to the field of e-commerce technology, and in particular to an information extraction method, apparatus, device, and medium. Background Technology

[0002] In the e-commerce sector, comprehensive, accurate, and timely access to product information is crucial for e-commerce platforms, data analytics companies, and market research institutions. Product information, such as titles, prices, images, and descriptions, not only influences consumers' purchasing decisions but also plays a key role in market trend analysis, competitor monitoring, and inventory management.

[0003] However, current product information extraction technologies face numerous challenges. Traditional web crawling methods have significant limitations in extracting product information in the e-commerce field. First, relying on technicians to manually specify the initial Uniform Resource Locator (URL) hinders the automation process and may lead to incomplete data crawling or the inclusion of irrelevant content, thus affecting crawling efficiency and data integrity. Second, manually writing crawling rules for specific website page structures not only increases system complexity and maintenance costs but also requires frequent adjustments when the website structure changes, reducing the system's adaptability and scalability.

[0004] Given the above, how to solve the problems faced by traditional web crawling methods in extracting product information, such as automation obstacles, incomplete data capture or inclusion of irrelevant content, increased system complexity, high maintenance costs, and reduced adaptability and scalability, is an urgent issue for technical personnel in this field. Summary of the Invention

[0005] The purpose of this application is to provide an information extraction method, apparatus, device, and medium to solve the problems faced by traditional web crawling methods in extracting product information, such as hindered automation, incomplete data capture or inclusion of irrelevant content, increased system complexity, high maintenance costs, and reduced adaptability and scalability.

[0006] To address the aforementioned technical problems, this application provides an information extraction method, comprising:

[0007] Multiple target sites are acquired, and the type of each target site is determined using rule-based hierarchical feature detection technology; the rules include at least priority hierarchical rules and feature fingerprint matching rules.

[0008] Obtain the links to the corresponding pages within each of the target sites based on their respective types;

[0009] Obtain the corresponding page based on each link, extract the XPath path of each tag in each page, and remove the subscript from each XPath path and classify and merge them to generate a classification dictionary corresponding to each page;

[0010] Determine the common paths corresponding to each of the classification dictionaries, and cluster each of the classification dictionaries according to the similarity of each of the common paths to generate the clustering results corresponding to each of the pages;

[0011] The clustering results are classified to obtain a target page containing the target information;

[0012] Extract the target information from each of the target pages.

[0013] On the one hand, acquire multiple target sites, including:

[0014] Determine the extraction requirements for the target information;

[0015] Determine the corresponding target sites based on the extraction requirements;

[0016] The target sites can be obtained through site collection websites or by searching through a browser.

[0017] On the other hand, the type of each target site is determined using rule-based hierarchical feature detection technology, including:

[0018] Obtain the website link of the target site, and add the JSON file path of the product information to the end of the website link to obtain the first website link corresponding to the target site;

[0019] Determine whether the first website link can be opened normally;

[0020] If it is confirmed that the first website link can be opened normally, then the type of the target site is confirmed to be a Shopify website, and the currency information and language information of the target site are extracted;

[0021] If it is confirmed that the first website link cannot be opened normally, a WordPress REST API product data request path is added to the end of the website link to obtain the second website link corresponding to the target site.

[0022] Determine whether the second website link can be opened normally;

[0023] If it is confirmed that the second website link can be opened normally, then the type of the target site is confirmed to be a WooCommerce website;

[0024] If it is confirmed that the second website link cannot be opened normally, a specified request is sent to the target site; wherein the request header of the specified request contains at least XMLHttpRequest, and the parameters of the specified request contain at least target=main and widget=XLite\View\Slidebar;

[0025] Determine whether navigation links can be extracted from the webpage content returned by the target site; wherein the navigation links contain the class="leaf";

[0026] If it is confirmed that the navigation link can be extracted, then the type of the target site is confirmed to be an X-Cart website;

[0027] If it is confirmed that the navigation link cannot be extracted, the website homepage of the target site is accessed directly, and it is determined whether each link on the website homepage contains a specified parameter; wherein, the specified parameter is route=product / category;

[0028] If the specified parameters are confirmed to be included, then the target site is confirmed to be an OpenCart website.

[0029] If it is confirmed that the specified parameter is not included, then the type of the target site is confirmed to be a non-SaaS site;

[0030] Among them, Shopify, WooCommerce, X-Cart, and OpenCart are all SaaS sites.

[0031] On the other hand, obtaining links to pages within the corresponding target sites based on their types includes:

[0032] When the target site is a SaaS site, access the specified API interface of the target site to obtain the link;

[0033] When the target site is a non-SaaS site, access the site map file of the target site and determine whether the site map file was accessed successfully.

[0034] If so, then parse each Uniform Resource Locator in the sitemap file to obtain the link;

[0035] If not, the links in the target site are crawled using a depth-first search crawler algorithm, and / or the pages of the target site are loaded using a web application automation testing tool, and the links in the pages are obtained.

[0036] On the other hand, based on each of the links, the corresponding page is obtained, the XPath path of each tag in each of the pages is extracted, and the XPath paths are deindexed, categorized, and merged, including:

[0037] Obtain the Hypertext Markup Language source code of the corresponding page based on each of the links;

[0038] Obtain the XPath paths of all tags under each of the aforementioned Hypertext Markup Language source codes;

[0039] The XPath paths described are categorized and merged after removing subscripts;

[0040] Correspondingly, the common paths corresponding to each of the classification dictionaries are determined, and the classification dictionaries are clustered and grouped according to the similarity of each of the common paths, including:

[0041] Based on the slash segmentation in each of the classification dictionaries, the common path corresponding to each of the classification dictionaries is determined;

[0042] The similarity of each of the common paths is calculated according to the Levenstein distance formula;

[0043] The common paths with the same similarity are identified as belonging to the same category, and an initial clustering result is generated.

[0044] Page clustering results are generated based on the initial clustering results and the DBSCAN density clustering algorithm.

[0045] Correspondingly, the clustering results are classified, including:

[0046] The page clustering results are classified using artificial intelligence technology to obtain a set of detail pages and a set of list pages.

[0047] On the other hand, extracting the target information from each target page includes:

[0048] Determine whether the source code of the detail pages in the set of detail pages contains structured data;

[0049] If so, extract the target information from the set of detail pages using a structured data extraction function;

[0050] If not, the page content in the collection of detail pages will be converted to Markdown format using Hypertext Markup Language to Markdown technology;

[0051] The target information in the page content in Markdown format is identified using artificial intelligence technology.

[0052] On the other hand, after identifying the target information in the page content in Markdown format using artificial intelligence technology, the method further includes:

[0053] Context-aware visual attention technology is used to extract relevant information corresponding to the target information in a simulated browser; wherein, the relevant information includes at least an XPath path;

[0054] The collection of detail pages is traversed according to the XPath path to extract the target information.

[0055] To address the aforementioned technical problems, this application also provides an information extraction device, comprising:

[0056] The first acquisition module is used to acquire multiple target sites and determine the type of each target site using rule-based hierarchical feature detection technology; the rules include at least priority hierarchical rules and feature fingerprint matching rules;

[0057] The second acquisition module is used to acquire links to pages in the corresponding target sites according to the type of each target site;

[0058] The first clustering module is used to obtain the corresponding page based on each link, extract the XPath path of each tag in each page, and perform subscript removal, classification and merging on each XPath path to generate a classification dictionary corresponding to each page.

[0059] The second clustering module is used to determine the common paths corresponding to each of the classification dictionaries, and to cluster each of the classification dictionaries according to the similarity of each of the common paths, so as to generate the clustering results corresponding to each of the pages;

[0060] A classification module is used to classify the clustering results to obtain a target page containing the target information;

[0061] The extraction module is used to extract the target information from each of the target pages.

[0062] To address the aforementioned technical problems, this application also provides an information extraction device, comprising:

[0063] Memory, used to store computer programs;

[0064] A processor is used to implement the steps of the information extraction method described above when executing the computer program.

[0065] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned information extraction method.

[0066] The information extraction method provided in this application automates the collection of target websites and utilizes rule-based hierarchical feature detection technology to determine the target website type, thereby reducing the number of searches, lowering traffic consumption, and improving recognition accuracy. It obtains links to pages within each target website based on their type, achieving efficient collection of product detail page links, thus reducing manual intervention and significantly improving the efficiency and accuracy of crawling on large-scale websites. Simultaneously, it clusters and classifies all pages corresponding to the links, automating page classification and recognition, and ultimately identifying the target information based on the target page. This solution, through automated and intelligent technical means, significantly improves the efficiency and quality of data crawling on large-scale e-commerce websites, meeting the needs of real-time data acquisition, and is particularly suitable for large-scale data crawling.

[0067] In addition, this application also provides an information extraction device, equipment and medium, with the same effect as above. Attached Figure Description

[0068] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 A flowchart illustrating an information extraction method provided in this application embodiment;

[0070] Figure 2 A schematic diagram of an information extraction device provided in an embodiment of this application;

[0071] Figure 3 This is a structural diagram of an information extraction device provided in an embodiment of this application. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0073] The core of this application is to provide an information extraction method, apparatus, device, and medium to solve the problems faced by traditional web crawling methods in extracting product information, such as hindered automation, incomplete data capture or inclusion of irrelevant content, increased system complexity, high maintenance costs, and reduced adaptability and scalability.

[0074] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] In the e-commerce sector, comprehensive, accurate, and timely access to product information is crucial for e-commerce platforms, data analytics companies, and market research institutions. Product information, such as titles, prices, images, and descriptions, not only influences consumers' purchasing decisions but also plays a key role in market trend analysis, competitor monitoring, and inventory management.

[0076] However, current product information extraction technologies face numerous challenges. Traditional web crawling methods rely on technicians manually specifying initial URLs, which is not only difficult to automate but may also lead to incomplete data capture or the inclusion of irrelevant content due to improper URL selection, thus affecting crawling efficiency and data integrity. Furthermore, manually writing crawling rules (such as XPath and CSS selectors) for specific website page structures increases system complexity and maintenance costs, and these rules need to be adjusted accordingly when the website structure changes, reducing the system's adaptability and scalability. For complex content dynamically loaded via JavaScript, traditional HTTP requests are insufficient, requiring advanced technologies such as rendering-based crawling.

[0077] In general, existing methods are inefficient for large-scale data crawling, failing to meet the demands of real-time data acquisition. Furthermore, manually written rules are prone to errors, affecting data accuracy. These issues collectively constrain the efficiency and quality of product information extraction. To address these problems, this application provides an information extraction method.

[0078] It should be noted that the application scenarios of the information extraction method provided in this application are not limited to extracting product information from e-commerce websites, but can also be used to extract news information from news websites, user-posted information from social media websites, and academic information from academic websites, depending on the specific implementation. Furthermore, it should be noted that the information extraction method provided in this application should be implemented in accordance with the terms of use of the websites involved, without affecting the normal operation of the websites, without infringing copyright, and in compliance with relevant data protection laws.

[0079] Figure 1 This is a flowchart illustrating an information extraction method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0080] S10: Obtain multiple target sites and determine the type of each target site using rule-based hierarchical feature detection technology. The rules include at least priority hierarchical rules and feature fingerprint matching rules.

[0081] Specifically, to extract target information, it is first necessary to acquire multiple target websites. This can be done through various channels, such as website aggregation sites and browser searches, to collect a large number of target websites. Simultaneously, the type of each target website needs to be determined.

[0082] It is important to note that this embodiment utilizes rule-based hierarchical feature detection technology to determine the type of each target site. The rules involved include at least priority hierarchical rules and feature fingerprint matching rules. Specifically, the priority hierarchical rules define the detection order and process jump logic; the feature fingerprint matching rules define the unique technical features and matching conditions for each platform. This embodiment does not limit the specific process for determining the type of each target site; nor does it limit the type of each target site. For example, it can be classified according to content type, function, software service model, technical architecture, device type, industry, etc., depending on the specific implementation.

[0083] S11: Obtain the links to the corresponding pages in each target site based on the type of each target site.

[0084] Furthermore, based on the type of each target site, corresponding methods are used to obtain links to pages within those sites, facilitating access to the corresponding pages via these links. It should be noted that this embodiment does not limit the specific process of obtaining links to pages within each target site based on its type; it depends on the specific implementation.

[0085] S12: Obtain the corresponding page based on each link, extract the XPath path of each tag in each page, and remove the subscript from each XPath path to classify and merge them to generate a category dictionary for each page.

[0086] S13: Determine the common paths corresponding to each category dictionary, and cluster each category dictionary according to the similarity of each common path to generate the clustering results corresponding to each page.

[0087] S14: Classify the clustering results to obtain the target page containing the target information.

[0088] Understandably, a website has many pages. After retrieving all pages, to determine their specific types, similar pages need to be grouped into the same category. Once a page in this category is detected, the entire category is considered to be of the same type. Similarly, if the XPath parsing rules for one page in this category are obtained, then all data in this category can be extracted using the same XPath. It's worth noting that XPath is a language used to locate and select nodes in Extensible Markup Language (XML) documents. It provides a powerful way to navigate and query XML structures, similar to using paths to locate files in a file system.

[0089] Therefore, in this embodiment, the corresponding page is obtained based on each link, the XPath path of each tag in each page is extracted, and the XPath paths are deindexed, classified, and merged to generate a classification dictionary for each page. The common paths corresponding to each classification dictionary are determined, and the classification dictionaries are clustered based on the similarity of the common paths to generate clustering results for each page. Simultaneously, each clustering result is classified to obtain the target page containing the target information.

[0090] It should be noted that this embodiment does not limit the clustering calculation process; for example, it can be K-means clustering or Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering. K-means clustering is a prototype-based clustering method that forms clusters by iteratively assigning data points to the nearest centroid. K-means is suitable for handling large-scale datasets and is typically used to identify and group similar data patterns. DBSCAN clustering is a density-based clustering method that can identify clusters of arbitrary shapes and handle noisy data. DBSCAN does not require pre-specifying the number of clusters and is suitable for handling complex data structures.

[0091] Furthermore, this embodiment does not impose any restrictions on the specific method of classifying each clustering result; it depends on the specific implementation situation.

[0092] S15: Extract target information from each target page.

[0093] Finally, the target information on each target page. It should be noted that this embodiment does not restrict the specific extraction process of the target information, nor does it limit the specific content of the target information. Taking product information extraction as an example, the target information should include product name, price, description, product image URL, product rating and reviews, product availability, and store information, etc.

[0094] In this embodiment, by automating the collection of target websites and utilizing rule-based hierarchical feature detection technology to determine the type of target websites, the number of searches can be reduced, traffic consumption can be lowered, and recognition accuracy can be improved. Links to pages within the corresponding target websites are obtained based on their types, achieving efficient collection of product detail page links, thereby reducing manual intervention and significantly improving the efficiency and accuracy of crawling on large-scale websites. Simultaneously, all pages corresponding to the links are clustered and classified, automating page classification and recognition, and ultimately identifying target information based on the target pages. This solution, through automated and intelligent technical means, significantly improves the efficiency and quality of data crawling on large-scale e-commerce websites, meeting the needs of real-time data acquisition, and is particularly suitable for large-scale data crawling.

[0095] Based on the above embodiments, in some embodiments, multiple target sites are acquired, including:

[0096] S100: Determine the extraction requirements for the target information.

[0097] S101: Determine the corresponding target sites based on the extraction requirements.

[0098] S102: Obtain target sites through site aggregation websites or by searching through a browser.

[0099] To obtain target websites, this embodiment first needs to determine the extraction requirements for the target information. For example, when it is necessary to obtain product information, it is necessary to obtain websites related to the product. Therefore, specific target websites are determined according to the extraction requirements. Subsequently, a large number of target websites are collected through website aggregation or browser search to facilitate the extraction of target information.

[0100] To determine the type of a target site, based on the above embodiments, in some embodiments, rule-based hierarchical feature detection technology is used to determine the type of each target site, including:

[0101] S103: Obtain the website link of the target site and add the JSON file path of the product information to the end of the website link to obtain the first website link corresponding to the target site.

[0102] S104: Determine whether the first website link can be opened normally; if yes, proceed to step S105; if no, proceed to step S106.

[0103] S105: Confirm that the target site is a Shopify website and extract the currency and language information of the target site.

[0104] S106: Add the WordPress REST API product data request path to the end of the website link to obtain a second website link corresponding to the target site.

[0105] S107: Determine whether the link to the second website can be opened normally; if yes, proceed to step S108; if no, proceed to step S109.

[0106] S108: Confirm that the target site is a WooCommerce website.

[0107] S109: Send a specified request to the target site.

[0108] The request headers of the specified request must contain at least XMLHttpRequest, and the parameters of the specified request must contain at least target=main and widget=XLite\View\Slidebar.

[0109] S110: Determine whether a navigation link can be extracted from the webpage content returned by the target site; wherein the navigation link contains class="leaf"; if yes, proceed to step S111. If no, proceed to step S112.

[0110] S111: Confirm that the target site is an X-Cart website.

[0111] S112: Directly access the target site's homepage and determine whether each link on the homepage contains the specified parameter; where the specified parameter is route=product / category; if yes, proceed to step S113; if no, proceed to step S114.

[0112] S113: Confirmed that the target site is an OpenCart website.

[0113] S114: Confirm that the target site is a non-SaaS site;

[0114] Among them, Shopify, WooCommerce, X-Cart, and OpenCart are all SaaS sites.

[0115] Specifically, first, obtain the target site's website link (e.g., https: / / example.com), and append the path to the product information's JSON file, i.e., / products.json, to the end of the website link to obtain the first website link corresponding to the target site (e.g., https: / / example.com / products.json). Determine if the first website link can be opened normally. If the first website link is confirmed to be open normally, then confirm that the target site is a Shopify website, and extract the target site's currency and language information. Specifically, search for the keyword "Shopify.currency" in the webpage source code and extract the currency code (e.g., USD) within the brackets. Simultaneously, search for "Shopify.locale" in the source code and extract the language code (e.g., en) within the quotation marks.

[0116] If the first website link is confirmed to be inaccessible, add the WordPress RESTAPI product data request path to the end of the website link, i.e., / wp-json / wp / v2 / product?page=1&per_page=100, to obtain the second website link corresponding to the target site. Check if the second website link can be opened normally; if it can, confirm that the target site is a WooCommerce website. If the second website link cannot be opened normally, send a specified request to the target site.

[0117] It should be noted that the request header must contain at least `XMLHttpRequest`, i.e., `x-requested-with: XMLHttpRequest`; and the request parameters must contain at least `target=main` and `widget=XLite\View\Slidebar`. The system checks if navigation links can be extracted from the webpage content returned by the target site; these navigation links must contain the class="leaf". If navigation links can be extracted, the target site is confirmed to be an X-Cart website.

[0118] If it is confirmed that navigation links cannot be extracted, then directly access the target site's homepage and check whether each link on the homepage contains the specified parameter; where the specified parameter is route=product / category. If the specified parameter is confirmed to be present, then the target site is confirmed to be an OpenCart website.

[0119] It should be noted that Shopify, WooCommerce, X-Cart, and OpenCart are all Software as a Service (SaaS) sites.

[0120] Furthermore, if the specified parameters are not found, confirm that the target site is a non-SaaS site. Further examine the descriptive information on the webpage (such as product details, titles, etc.) and use a translation tool to determine the language type. If there is no description, extract keywords (such as `language:en`) from the hidden data in the webpage code. Simultaneously scan the amount format in the webpage code, or the value after the keyword "currency" (such as "USD"). If all the above steps fail, access the ` / robots.txt` file, find the `sitemap.xml` link, and directly scrape the content from the sitemap.

[0121] Based on the above embodiments, in some embodiments, links to pages in the corresponding sites are obtained according to the type of each target site, including:

[0122] S115 When the target site is a SaaS site, access the specified API interface of the target site to obtain a link;

[0123] S116: When the target site is not a SaaS site, access the site map file of the target site and determine whether the site map file was accessed successfully; if yes, proceed to step S117; if no, proceed to step S118.

[0124] S117: Parse the Uniform Resource Locators in the site map file to obtain the links.

[0125] S118: Use a depth-first search crawler algorithm to crawl links in the target site, and / or use a web application automation testing tool to load pages in the target site and obtain the links in the pages.

[0126] Specifically, when the target site is a SaaS site, access the specified API interface of the target site to obtain the link. For example, access the wp-json / wp / v2 / product interface to obtain links to detailed information for all products.

[0127] When the target site is not a SaaS site, the system accesses the sitemap (sitemap.xml) file and checks if the sitemap file was accessed successfully. If so, it parses the Uniform Resource Locators (URLs) in the sitemap file to obtain the links. If not, it uses a Depth-First Search (DFS) crawler algorithm to recursively access all links on the page, starting from the initial URL, until all product detail page links are crawled, or crawling stops when a certain threshold is reached. It's important to note that the initial URL needs to be manually specified by technical personnel; the system automatically recursively crawls all links on the page. Additionally, web application automation testing tools (such as Selenium) can simulate a browser, load pages on the target site, process dynamically loaded data, and save the source code and page screenshots to obtain links from the page.

[0128] In this way, by employing multiple strategies to automatically collect initial URLs and all product detail page links, it can adapt to e-commerce sites of different types and sizes.

[0129] To achieve page clustering and classification, based on the above embodiments, in some embodiments, the corresponding page is obtained according to each link, the XPath path of each tag in each page is extracted, and the XPath paths are deindexed, classified, and merged, including:

[0130] S121: Obtain the Hypertext Markup Language source code of the corresponding page based on each link.

[0131] S122: Get the XPath path of all tags in each Hypertext Markup Language source code.

[0132] S123: Perform subscript removal, classification, and merging on each XPath path.

[0133] Specifically, for each collected link, a HyperText Markup Language (HTML) request is sent or the page content of the corresponding page is obtained through a rendering crawler. The page content must contain at least the HTML source code.

[0134] Furthermore, obtain all the HTML source code. The XPath of the tags is used to categorize and merge the XPaths, removing their indices, to generate a category dictionary. It's important to note that the key of the category dictionary is the XPath with its indices removed, and the value is a list of the original XPaths. For example:

[0135] {

[0136] " / html / body / div / div / a" : [

[0137] " / html / body / div / div[1] / a",

[0138] " / html / body / div / div[2] / a",

[0139] " / html / body / div / div[3] / a"

[0140] ],

[0141] " / html / body / table / tr / td / a" : [

[0142] " / html / body / table / tr[1] / td[1] / a" ,

[0143] " / html / body / table / tr[1] / td[2] / a" ,

[0144] " / html / body / table / tr[1] / td[3] / a" ,

[0145] " / html / body / table / tr[2] / td[1] / a" ,

[0146] " / html / body / table / tr[2] / td[2] / a" ,

[0147] " / html / body / table / tr[2] / td[3] / a" ]

[0149] }

[0150] Based on this, in some embodiments, common paths corresponding to each category dictionary are determined, and the category dictionaries are clustered and grouped according to the similarity of each common path, including:

[0151] S124: Based on the slash segmentation in each category dictionary, determine the common path corresponding to each category dictionary.

[0152] S125: Calculate the similarity of each common path according to the Levenstein distance formula.

[0153] S126: Identify common paths with the same similarity as belonging to the same category and generate the initial clustering results.

[0154] S127: Generate page clustering results based on the initial clustering results and the DBSCAN density clustering algorithm.

[0155] Specifically, based on the list of category dictionaries and the segmentation based on the forward slash ( / ) character, the common path corresponding to each category dictionary is determined, thereby converting the category dictionaries into a list, as follows:

[0156] {

[0157] " / html / body / div": 1,

[0158] " / html / body / table": 1

[0159] }

[0160] [" / html / body / div"," / html / body / table"];

[0161] Understandably, after the above steps, each page on each website becomes a categorization dictionary. Finally, the similarity of each common path is calculated using the Levenshtein distance formula. The Levenshtein distance measures the difference between two strings and is defined as the minimum number of editing operations (including insertion, deletion, and replacement characters) required to transform one string into another. The specific formula is as follows:

[0162] ;

[0163] Where a and b are two strings, i and j represent the current character position being processed; cost takes the value: if a i Not equal to b j If the value is 1, then the result is 1; otherwise, it is 0. For example, the Levenstein distance between the strings "kitten" and "sitting" is 3. The conversion process is as follows: kitten → sitten (replace the first character k with s), sitten → sittin (replace the fifth character e with i), sittin → sitting (insert the character g at the end).

[0164] After calculating the similarity of each common path according to the Levenshtein formula, common paths with the same similarity are grouped into the same category, generating initial clustering results. Finally, page clustering results are generated based on the initial clustering results and the DBSCAN density clustering algorithm.

[0165] Correspondingly, the clustering results are categorized, including:

[0166] S128: Classify the page clustering results using artificial intelligence technology to obtain a set of detail pages and a set of list pages.

[0167] The clustering results are categorized using Artificial Intelligence (AI) technology to classify the clustered pages, resulting in a set of detail pages and a set of list pages. It's important to note that the detail page set contains multiple detail pages, and the list page set contains multiple list pages; each detail page typically corresponds to information about a single product. These pages provide detailed information about the product, including description, specifications, price, images, and user reviews. A detail page usually corresponds to a specific product. List pages display brief information about multiple products. These pages typically include product thumbnails, names, prices, etc., for users to browse and select. List pages are used to display a collection of products from which users can select one to view its detail page. Therefore, typically one product corresponds to one detail page, while one list page corresponds to brief information about multiple products. The list page serves as an entry point; users can click on product links to access the corresponding detail page for more detailed information.

[0168] It should be noted that this embodiment does not limit the specific AI technology used. For example, it can be a large language model (LLM), a machine learning model, including traditional machine learning models such as decision trees, random forests or support vector machines (SVM), or a deep learning model, including convolutional neural networks or other deep learning architectures, which can be used for more complex feature extraction and classification tasks, especially when dealing with large amounts of unstructured data.

[0169] In this embodiment, a clustering algorithm is used to cluster all links, and AI technology is used to sample and identify each category, thereby intelligently distinguishing list pages and detail pages, and automating page classification and identification in large-scale websites.

[0170] In order to extract target information, based on the above embodiments, in some embodiments, target information is extracted from each target page, including:

[0171] S131: Determine whether there is structured data in the source code of the detail pages in the detail page collection; if yes, proceed to step S132; if no, proceed to step S133.

[0172] S132: Extract target information from the details page collection using structured data extraction functions.

[0173] S133: Convert the page content in the detail page collection into Markdown format using Hypertext Markup Language to Markdown technology.

[0174] S134: Identify target information in Markdown formatted page content using artificial intelligence technology.

[0175] Specifically, it determines whether structured data exists in the source code of the detail pages in the detail page collection. If structured data exists, the target information in the detail page collection is extracted using the structured data extraction function `extruct`.

[0176] It's important to note that `extruct` is a Python library for extracting structured data from web pages. Structured data is typically embedded in the web page source code in a specific format, making the data easier for machines to read and process. Common structured data formats include JSON-LD, Microdata, and RDFa. These formats are often used to embed semantic information about the page content so that search engines and other tools can better understand and display the data. Therefore, using `extruct` to extract this structured data can yield detailed information about the web page content, typically including product names, prices, descriptions, product image URLs, product ratings and reviews, product availability, and store information.

[0177] If structured data is not available, the page content in the detail page collection is converted into Markdown format using HTML to Markdown technology. AI technology is then used to identify target information in the Markdown format page content, including but not limited to product titles, prices, and images.

[0178] In this embodiment, by combining AI technology and HTML to Markdown technology, the page content can be intelligently parsed to identify information such as product titles, prices, and images.

[0179] To avoid AI technology failing to extract all target information, based on the above embodiments, some embodiments further include, after identifying the target information in the Markdown format page content using artificial intelligence technology:

[0180] S135: Extract relevant information corresponding to the target information in a simulated browser using context-aware visual attention technology.

[0181] The relevant information includes at least the XPath path.

[0182] S136: Traverse the collection of detail pages according to the XPath path to extract the target information.

[0183] Specifically, if the AI ​​model fails to accurately extract all information, context-aware visual attention (CoVA) technology is used to extract relevant information. This relevant information includes at least the XPath paths of the product title, price, and images. A simulated browser is used to take screenshots of the page, and then CoVA is used to extract specific fields and their locations. CoVA itself provides the ability to identify and locate web page elements, while XPath is the specific technique used to identify the location of these elements. The combination of the two enables efficient web page data extraction. Finally, the set of detail pages is traversed according to the XPath paths to extract target information such as the titles, prices, and images of all products.

[0184] In this embodiment, when AI technology cannot accurately identify the field, the Cova tool is used to extract the XPath path of the corresponding field and automatically generate parsing rules, which improves the system's adaptability to large-scale sites and reduces maintenance costs and complexity.

[0185] In the above embodiments, the information extraction method has been described in detail. This application also provides embodiments of the information extraction device.

[0186] Figure 2 This is a schematic diagram of an information extraction device provided in an embodiment of this application. Figure 2 As shown, the device includes:

[0187] The first acquisition module 10 is used to acquire multiple target sites and determine the type of each target site using rule-based hierarchical feature detection technology; the rules include at least priority hierarchical rules and feature fingerprint matching rules;

[0188] The second acquisition module 11 is used to obtain the links of the pages in the corresponding site according to the type of each target site;

[0189] The first clustering module 12 is used to obtain the corresponding page based on each link, extract the XPath path of each tag in each page, and perform subscript removal, classification and merging of each XPath path to generate a classification dictionary corresponding to each page.

[0190] The second clustering module 13 is used to determine the common paths corresponding to each category dictionary, and to cluster each category dictionary according to the similarity of each common path to generate the clustering results corresponding to each page.

[0191] Classification module 14 is used to classify the clustering results to obtain the target page containing the target information;

[0192] Extraction module 15 is used to extract target information from each target page.

[0193] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0194] Figure 3 This is a structural diagram of an information extraction device provided in an embodiment of this application. Figure 3 As shown, the information extraction device includes:

[0195] Memory 20 is used to store computer programs;

[0196] The processor 21 is used to implement the steps of the information extraction method mentioned in the above embodiments when executing a computer program.

[0197] The information extraction device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0198] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0199] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the information extraction method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the information extraction method.

[0200] In some embodiments, the information extraction device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0201] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the information extraction device and may include more or fewer components than illustrated.

[0202] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0203] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] The foregoing has provided a detailed description of an information extraction method, apparatus, device, and medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0205] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. An information extraction method, characterized in that, include: Multiple target sites are acquired, and the type of each target site is determined using rule-based hierarchical feature detection technology; the rules include at least priority hierarchical rules and feature fingerprint matching rules. Obtain the links to the corresponding pages within each of the target sites based on their respective types; Obtain the corresponding page based on each link, extract the XPath path of each tag in each page, and remove the subscript from each XPath path and classify and merge them to generate a classification dictionary corresponding to each page; Determine the common paths corresponding to each of the classification dictionaries, and cluster each of the classification dictionaries according to the similarity of each of the common paths to generate the clustering results corresponding to each of the pages; The clustering results are classified to obtain the target page containing the target information; Extract the target information from each of the target pages; Based on the links provided, obtain the corresponding pages, extract the XPath paths of each tag on each page, and then remove the indices from the XPath paths, categorize and merge them, including: Obtain the Hypertext Markup Language source code of the corresponding page based on each of the links; Obtain the XPath paths of all tags under each of the aforementioned Hypertext Markup Language source codes; The XPath paths described are categorized and merged after removing subscripts; Correspondingly, the common paths corresponding to each of the classification dictionaries are determined, and the classification dictionaries are clustered and grouped according to the similarity of each of the common paths, including: Based on the slash segmentation in each of the classification dictionaries, the common path corresponding to each of the classification dictionaries is determined; The similarity of each of the common paths is calculated according to the Levenstein distance formula; The common paths with the same similarity are identified as belonging to the same category, and an initial clustering result is generated. Page clustering results are generated based on the initial clustering results and the DBSCAN density clustering algorithm. Correspondingly, the clustering results are classified, including: The page clustering results are classified using artificial intelligence technology to obtain a set of detail pages and a set of list pages.

2. The information extraction method according to claim 1, characterized in that, Obtain multiple target sites, including: Determine the extraction requirements for the target information; Determine the corresponding target sites based on the extraction requirements; The target sites can be obtained through site collection websites or by searching through a browser.

3. The information extraction method according to claim 1, characterized in that, The type of each target site is determined using rule-based hierarchical feature detection technology, including: Obtain the website link of the target site, and add the JSON file path of the product information to the end of the website link to obtain the first website link corresponding to the target site; Determine whether the first website link can be opened normally; If it is confirmed that the first website link can be opened normally, then the type of the target site is confirmed to be a Shopify website, and the currency information and language information of the target site are extracted; If it is confirmed that the first website link cannot be opened normally, a WordPress REST API product data request path is added to the end of the website link to obtain the second website link corresponding to the target site. Determine whether the second website link can be opened normally; If it is confirmed that the second website link can be opened normally, then the type of the target site is confirmed to be a WooCommerce website; If it is confirmed that the second website link cannot be opened normally, a specified request is sent to the target site; wherein the request header of the specified request contains at least XMLHttpRequest, and the parameters of the specified request contain at least target=main and widget=XLite\View\Slidebar; Determine whether navigation links can be extracted from the webpage content returned by the target site; wherein the navigation links contain the class="leaf"; If it is confirmed that the navigation link can be extracted, then the type of the target site is confirmed to be an X-Cart website; If it is confirmed that the navigation link cannot be extracted, the website homepage of the target site is accessed directly, and it is determined whether each link on the website homepage contains a specified parameter; wherein, the specified parameter is route=product / category; If the specified parameters are confirmed to be included, then the target site is confirmed to be an OpenCart website. If it is confirmed that the specified parameter is not included, then the type of the target site is confirmed to be a non-SaaS site; Among them, Shopify, WooCommerce, X-Cart, and OpenCart are all SaaS sites.

4. The information extraction method according to claim 3, characterized in that, Obtain links to pages within the corresponding target sites based on their types, including: When the target site is a SaaS site, access the specified API interface of the target site to obtain the link; When the target site is a non-SaaS site, access the site map file of the target site and determine whether the site map file was accessed successfully. If so, then parse each Uniform Resource Locator in the sitemap file to obtain the link; If not, the links in the target site are crawled using a depth-first search crawler algorithm, and / or the pages of the target site are loaded using a web application automation testing tool, and the links in the pages are obtained.

5. The information extraction method according to claim 1, characterized in that, Extracting the target information from each of the target pages, including: Determine whether the source code of the detail pages in the set of detail pages contains structured data; If so, extract the target information from the set of detail pages using a structured data extraction function; If not, the page content in the collection of detail pages will be converted to Markdown format using Hypertext Markup Language to Markdown technology; The target information in the page content in Markdown format is identified using artificial intelligence technology.

6. The information extraction method according to claim 5, characterized in that, After identifying the target information in the Markdown formatted page content using artificial intelligence technology, the process also includes: Context-aware visual attention technology is used to extract relevant information corresponding to the target information in a simulated browser; wherein, the relevant information includes at least an XPath path; The collection of detail pages is traversed according to the XPath path to extract the target information.

7. An information extraction device, characterized in that, include: The first acquisition module is used to acquire multiple target sites and determine the type of each target site using rule-based hierarchical feature detection technology; the rules include at least priority hierarchical rules and feature fingerprint matching rules; The second acquisition module is used to acquire links to pages in the corresponding target sites according to the type of each target site; The first clustering module is used to obtain the corresponding page based on each link, extract the XPath path of each tag in each page, and perform subscript removal, classification and merging on each XPath path to generate a classification dictionary corresponding to each page. The second clustering module is used to determine the common paths corresponding to each of the classification dictionaries, and to cluster each of the classification dictionaries according to the similarity of each of the common paths, so as to generate the clustering results corresponding to each of the pages; The classification module is used to classify the clustering results to obtain the target page containing the target information; The extraction module is used to extract the target information from each of the target pages; Based on the links provided, obtain the corresponding pages, extract the XPath paths of each tag on each page, and then remove the indices from the XPath paths, categorize and merge them, including: Obtain the Hypertext Markup Language source code of the corresponding page based on each of the links; Obtain the XPath paths of all tags under each of the aforementioned Hypertext Markup Language source codes; The XPath paths described are categorized and merged after removing subscripts; Correspondingly, the common paths corresponding to each of the classification dictionaries are determined, and the classification dictionaries are clustered and grouped according to the similarity of each of the common paths, including: Based on the slash segmentation in each of the classification dictionaries, the common path corresponding to each of the classification dictionaries is determined; The similarity of each of the common paths is calculated according to the Levenstein distance formula; The common paths with the same similarity are identified as belonging to the same category, and an initial clustering result is generated. Page clustering results are generated based on the initial clustering results and the DBSCAN density clustering algorithm. Correspondingly, the clustering results are classified, including: The page clustering results are classified using artificial intelligence technology to obtain a set of detail pages and a set of list pages.

8. An information extraction device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the information extraction method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the information extraction method as described in any one of claims 1 to 6.

Images