Information extraction method and device, equipment and medium

Abstract

The invention discloses an information extraction method and device, equipment and a medium, and relates to the technical field of electronic commerce. According to the scheme, the target sites are automatically collected, and the types of the target sites are determined by using a rule-based hierarchical feature detection technology, so that the retrieval times can be reduced, the flow consumption can be reduced, and the recognition accuracy can be improved; the links of the pages in the corresponding sites are obtained according to the types of the target sites, efficient collection of the links of the commodity detail pages is achieved, manual intervention of personnel is reduced, and the grabbing efficiency and accuracy in large-scale sites are greatly improved; meanwhile, the pages corresponding to all the links are clustered and classified and recognized, automation of page classification and recognition is achieved, and finally the target information is recognized based on the target page. According to the scheme, through automatic and intelligent technical means, the data capturing efficiency and quality in large-scale e-commerce sites are greatly improved, the requirement for real-time data acquisition can be met, and the method is particularly suitable for large-scale data capturing.

Description

Technical Field

[0001] The present application relates to the field of e-commerce technology, and in particular to an information extraction method, device, equipment and medium. Background Art

[0002] In the e-commerce sector, comprehensive, accurate, and timely access to product information is crucial for e-commerce platforms, data analytics companies, and market research organizations. Product information, such as titles, prices, images, and descriptions, not only influences consumer purchasing decisions but also plays a key role in market trend analysis, competitor monitoring, and inventory management.

[0003] However, current product information extraction technology faces numerous challenges. Traditional crawling methods have significant limitations for extracting product information in the e-commerce sector. First, relying on technicians to manually specify the initial Uniform Resource Locator (URL) hinders automation and can result in incomplete data capture or the inclusion of irrelevant content, compromising crawling efficiency and data integrity. Second, manually writing crawling rules tailored to specific website page structures not only increases system complexity and maintenance costs, but also requires frequent adjustments when the website structure changes, reducing the system's adaptability and scalability.

[0004] In view of the above, how to solve the problems faced by traditional crawler methods in extracting product information, such as automation obstruction, incomplete data capture or inclusion of irrelevant content, increased system complexity, high maintenance costs, and reduced adaptability and scalability, is an urgent problem that needs to be solved by technical personnel in this field. Summary of the Invention

[0005] The purpose of this application is to provide an information extraction method, device, equipment and medium to solve the problems faced by traditional crawler methods in product information extraction, such as automation obstruction, incomplete data capture or inclusion of irrelevant content, increased system complexity, high maintenance costs, and reduced adaptability and scalability.

[0006] To solve the above technical problems, the present application provides an information extraction method, comprising:

[0007] Acquire multiple target sites and determine the type of each target site using a rule-based hierarchical feature detection technology; the rule includes at least a priority hierarchical rule and a feature fingerprint matching rule;

[0008] Obtaining a link to a page in the corresponding site according to the type of each target site;

[0009] Obtaining a corresponding page according to each link, extracting the XPath path of each tag in each page, and removing subscripts and classifying and merging each XPath path to generate a classification dictionary corresponding to each page;

[0010] Determining common paths corresponding to the classification dictionaries, and clustering and grouping the classification dictionaries according to similarities of the common paths to generate clustering results corresponding to the pages;

[0011] Classifying each of the clustering results to obtain a target page containing the target information;

[0012] The target information in each target page is extracted.

[0013] On the one hand, multiple target sites are acquired, including:

[0014] Determining the need to extract the target information;

[0015] Determining the corresponding target sites according to the extraction requirements;

[0016] The target sites are obtained through a site collection website or through a browser search.

[0017] On the other hand, the type of each target site is determined using a rule-based hierarchical feature detection technology, including:

[0018] Obtain the website link of the target site, and add the JSON file path of the product information to the end of the website link to obtain the first website link corresponding to the target site;

[0019] Determining whether the first website link can be opened normally;

[0020] If it is confirmed that the first website link can be opened normally, then confirming that the type of the target site is a Shopify website, and extracting the currency information and language information of the target site;

[0021] If it is confirmed that the first website link cannot be opened normally, then add the WordPress REST API product data request path to the end of the website link to obtain a second website link corresponding to the target site;

[0022] Determining whether the second website link can be opened normally;

[0023] If it is confirmed that the second website link can be opened normally, then the type of the target site is confirmed to be a WooCommerce website;

[0024] If it is confirmed that the second website link cannot be opened normally, a specified request is sent to the target site; wherein the request header of the specified request at least includes XMLHttpRequest, and the parameters of the specified request at least include target=main and widget=XLite\View\Slidebar;

[0025] Determine whether a navigation link can be extracted from the webpage content returned by the target site; wherein the navigation link has class="leaf";

[0026] If it is confirmed that the navigation link can be extracted, then the type of the target site is confirmed to be an X-Cart website;

[0027] If it is confirmed that the navigation link cannot be extracted, directly access the homepage of the target site and determine whether each link on the homepage contains a specified parameter; wherein the specified parameter is route=product / category;

[0028] If it is confirmed that the specified parameters are included, then the type of the target site is confirmed to be an OpenCart website;

[0029] If it is confirmed that the specified parameter is not included, then the type of the target site is confirmed to be a non-SaaS site;

[0030] Among them, Shopify website, WooCommerce website, X-Cart website, and OpenCart website are all SaaS sites.

[0031] On the other hand, obtaining a link to a page in each target site according to the type of the target site includes:

[0032] When the target site is a SaaS site, accessing a designated API interface of the target site to obtain the link;

[0033] When the target site is a non-SaaS site, accessing a website map file of the target site to determine whether the website map file is successfully accessed;

[0034] If so, parsing each uniform resource locator in the sitemap file to obtain the link;

[0035] If not, the link in the target site is crawled by a depth-first search crawler algorithm, and / or the page of the target site is loaded by a Web application automation testing tool, and the link in the page is obtained.

[0036] On the other hand, the corresponding page is obtained according to each link, the XPath path of each tag in each page is extracted, and each XPath path is removed from the subscript, classified and merged, including:

[0037] Obtaining the Hypertext Markup Language source code of the corresponding page according to each link;

[0038] Obtain the XPath paths of all a tags under each of the hypertext markup language source codes;

[0039] Remove subscripts from each XPath path, classify and merge;

[0040] Correspondingly, determining the common paths corresponding to the classification dictionaries and clustering the classification dictionaries according to the similarities of the common paths includes:

[0041] Determining the common path corresponding to each classification dictionary based on the slash character segmentation in each classification dictionary;

[0042] Calculating the similarity of each of the common paths according to the Levenshtein distance formula;

[0043] Determine the common paths corresponding to the same similarity as the same type, and generate an initial clustering result;

[0044] Generate page clustering results based on the initial clustering results and the DBSCAN density clustering algorithm;

[0045] Correspondingly, each clustering result is classified, including:

[0046] The page clustering results are classified according to artificial intelligence technology to obtain a details page set and a list page set.

[0047] On the other hand, extracting the target information from each target page includes:

[0048] Determine whether structured data exists in the detail page source code of the detail page set;

[0049] If so, extract the target information from the detail page set using a structured data extraction function;

[0050] If not, convert the page content in the detail page set into Markdown format using Hypertext Markup Language to Markdown technology;

[0051] The target information in the page content in Markdown format is identified through artificial intelligence technology.

[0052] On the other hand, after identifying the target information in the page content in Markdown format using artificial intelligence technology, the method further includes:

[0053] Extracting relevant information corresponding to the target information in a simulated browser by using context-aware visual attention technology; wherein the relevant information at least includes an XPath path;

[0054] The detail page set is traversed according to the XPath path to extract the target information.

[0055] To solve the above technical problems, the present application also provides an information extraction device, comprising:

[0056] A first acquisition module is configured to acquire a plurality of target sites and determine the type of each target site using a rule-based hierarchical feature detection technique; the rule includes at least a priority hierarchical rule and a feature fingerprint matching rule;

[0057] A second acquisition module is used to acquire a link to a page in each target site according to the type of the target site;

[0058] A first clustering module is configured to obtain a corresponding page according to each link, extract the XPath path of each tag in each page, and perform classification and merging on each XPath path without subscripts to generate a classification dictionary corresponding to each page;

[0059] A second clustering module is used to determine the common paths corresponding to the classification dictionaries, and cluster the classification dictionaries according to the similarity of the common paths to generate clustering results corresponding to the pages;

[0060] A classification module, configured to classify each of the clustering results to obtain a target page containing the target information;

[0061] The extraction module is used to extract the target information from each target page.

[0062] To solve the above technical problems, the present application also provides an information extraction device, comprising:

[0063] memory for storing computer programs;

[0064] A processor is used to implement the steps of the above-mentioned information extraction method when executing the computer program.

[0065] In order to solve the above technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above information extraction method are implemented.

[0066] The information extraction method provided in this application can reduce the number of searches, reduce traffic consumption, and improve recognition accuracy by automatically collecting target sites and using rule-based hierarchical feature detection technology to determine the target site type; it obtains links to pages in the corresponding site according to the type of each target site, and realizes the efficient collection of product detail page links, thereby reducing manual intervention by personnel and greatly improving the crawling efficiency and accuracy in large-scale sites; at the same time, the pages corresponding to all links are clustered and classified and identified, realizing the automation of page classification and identification, and finally identifying the target information based on the target page. This solution greatly improves the efficiency and quality of data crawling in large-scale e-commerce sites through automated and intelligent technical means, can meet the needs of real-time data acquisition, and is particularly suitable for large-scale data crawling.

[0067] In addition, the present application also provides an information extraction device, equipment and medium, with the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 A flowchart of an information extraction method provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of an information extraction device provided in an embodiment of the present application;

[0071] Figure 3 A structural diagram of an information extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0073] The core of this application is to provide an information extraction method, device, equipment and medium to solve the problems faced by traditional crawler methods in product information extraction, such as automation obstruction, incomplete data capture or inclusion of irrelevant content, increased system complexity, high maintenance cost, and reduced adaptability and scalability.

[0074] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0075] In the e-commerce sector, comprehensive, accurate, and timely access to product information is crucial for e-commerce platforms, data analytics companies, and market research organizations. Product information, such as titles, prices, images, and descriptions, not only influences consumer purchasing decisions but also plays a key role in market trend analysis, competitor monitoring, and inventory management.

[0076] However, current product information extraction technology faces numerous challenges. Traditional crawling methods rely on technicians to manually specify the initial URL, which is not only difficult to automate but can also lead to incomplete data capture or the inclusion of irrelevant content due to improper URL selection, thus compromising crawling efficiency and data integrity. Furthermore, manually writing crawling rules (such as XPath and CSS selectors) tailored to specific website page structures increases system complexity and maintenance costs. Changes to the website structure also require adjustments to the rules, reducing the system's adaptability and scalability. Complex content dynamically loaded via JavaScript is difficult to retrieve with traditional HTTP requests and requires the use of advanced techniques such as rendering-based crawlers.

[0077] Overall, existing methods are inefficient for large-scale data capture, struggle to meet real-time data acquisition requirements, and manually written rules are prone to errors, affecting data accuracy. These issues collectively restrict the efficiency and quality of product information extraction. To address these issues, this application provides an information extraction method.

[0078] It should be noted that the application scenarios of the information extraction method provided in this application are not limited to the extraction of product information from product websites. It can also be used to extract news information from news websites, user-posted information from social media websites, and academic information from academic websites, depending on the specific implementation situation. In addition, it should be noted that the information extraction method provided in this application should be implemented on the basis of complying with the terms of use of the websites involved, not affecting the normal operation of the websites, not infringing on copyrights, and complying with relevant data protection laws.

[0079] Figure 1 This is a flow chart of an information extraction method provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0080] S10: Acquire multiple target sites and determine the type of each target site using a rule-based hierarchical feature detection technology. The rules include at least a priority hierarchical rule and a feature fingerprint matching rule.

[0081] Specifically, to extract target information, we first need to obtain multiple target sites. This can be done through various channels, such as site aggregation websites and browser searches. Furthermore, we need to determine the type of each target site.

[0082] It should be noted that in this embodiment, a rule-based hierarchical feature detection technology is used to determine the type of each target site, and the rules involved include at least priority hierarchical rules and feature fingerprint matching rules. Among them, the priority hierarchical rules specifically define the detection order and process jump logic; the feature fingerprint matching rules specifically define the unique technical features and matching conditions of each platform. In this embodiment, there is no restriction on the specific process of determining the type of each target site; there is no restriction on the type of each target site. For example, it can be classified according to content type, function, software service model, technical architecture, device type, industry, etc., depending on the specific implementation situation.

[0083] S11: Obtain links to pages in corresponding sites according to the types of the target sites.

[0084] Furthermore, according to the type of each target site, a corresponding method is used to obtain a link to a page in the corresponding site, so that the corresponding page can be accessed according to the link. It should be noted that the specific process of obtaining a link to a page in the corresponding site according to the type of each target site is not limited in this embodiment and depends on the specific implementation situation.

[0085] S12: Obtain the corresponding page according to each link, extract the XPath path of each tag in each page, and remove the subscripts, classify and merge each XPath path to generate a classification dictionary corresponding to each page.

[0086] S13: determining the common paths corresponding to the classification dictionaries, and clustering and grouping the classification dictionaries according to the similarities of the common paths to generate clustering results corresponding to the pages.

[0087] S14: Classify each clustering result to obtain a target page containing target information.

[0088] It's understandable that a website has many pages. After acquiring all the pages, in order to determine the specific types of the pages, it's necessary to group similar pages into the same category. As long as one page of this category is detected, all pages in this category are of the same type. Similarly, if the XPath parsing rule for one page of this category is obtained, then all data in this category can be extracted using the same XPath. It should be noted that XPath is a language used to locate and select nodes in Extensible Markup Language (XML) documents. It provides a powerful method for navigating and querying XML structures, similar to using paths to locate files in a file system.

[0089] Therefore, in this embodiment, the corresponding page is obtained based on each link. The XPath path of each tag in each page is extracted and then categorized and merged without subscripts to generate a classification dictionary corresponding to each page. The common paths corresponding to each classification dictionary are determined, and the classification dictionaries are clustered and grouped based on the similarity of the common paths to generate clustering results corresponding to each page. Simultaneously, each clustering result is classified to obtain the target page containing the target information.

[0090] It should be noted that the present embodiment does not limit the clustering calculation process, for example, it can be K-means clustering or density-based spatial clustering of applications with noise (DBSCAN) clustering. K-means clustering is a prototype-based clustering method that forms clusters by iteratively assigning data points to the nearest centroid. K-means is suitable for processing large-scale data sets and is typically used to identify and group similar data patterns. DBSCAN clustering is a density-based clustering method that can identify clusters of arbitrary shapes and process noisy data. DBSCAN does not require the number of clusters to be specified in advance and is suitable for processing complex data structures.

[0091] In addition, in this embodiment, there is no limitation on the specific method of classifying the clustering results, which depends on the specific implementation situation.

[0092] S15: Extracting target information from each target page.

[0093] Finally, the target information on each target page. It should be noted that this embodiment does not restrict the specific process for extracting target information, nor does it restrict the specific content of the target information. Taking product information extraction as an example, the target information should include product name, price, description, product image URL, product ratings and reviews, product availability, and store information.

[0094] In this embodiment, by automatically collecting target sites and using rule-based hierarchical feature detection technology to determine the target site type, the number of searches can be reduced, traffic consumption can be reduced, and recognition accuracy can be improved; links to pages in the corresponding site are obtained according to the type of each target site, achieving efficient collection of links to product detail pages, thereby reducing manual intervention by personnel and significantly improving the efficiency and accuracy of crawling in large-scale sites; at the same time, the pages corresponding to all links are clustered and classified and identified, achieving automation of page classification and identification, and ultimately identifying target information based on the target page. This solution significantly improves the efficiency and quality of data crawling in large-scale e-commerce sites through automated and intelligent technical means, can meet the needs of real-time data acquisition, and is particularly suitable for large-scale data crawling.

[0095] Based on the above embodiment, in some embodiments, obtaining multiple target sites includes:

[0096] S100: Determine the extraction requirements for target information.

[0097] S101: Determine corresponding target sites according to extraction requirements.

[0098] S102: Acquire each target site through a site collection website or through a browser search.

[0099] To obtain target sites, in this embodiment, the first step is to determine the target information extraction requirements. For example, when obtaining product information, sites related to the product are required. Therefore, the corresponding target sites are determined based on the extraction requirements. Subsequently, a large number of target sites are collected through site aggregation websites or browser retrieval to facilitate the extraction of target information.

[0100] In order to determine the type of the target site, based on the above embodiments, in some embodiments, a rule-based hierarchical feature detection technology is used to determine the type of each target site, including:

[0101] S103: Obtain a website link of the target site, and add a JSON file path of the product information to the end of the website link to obtain a first website link corresponding to the target site.

[0102] S104: Determine whether the first website link can be opened normally; if so, proceed to step S105; if not, proceed to step S106.

[0103] S105: Confirm that the type of the target site is a Shopify website, and extract the currency information and language information of the target site.

[0104] S106: Add the WordPress REST API product data request path to the end of the website link to obtain a second website link corresponding to the target site.

[0105] S107: Determine whether the second website link can be opened normally; if so, proceed to step S108; if not, proceed to step S109.

[0106] S108: Confirm that the target site type is a WooCommerce website.

[0107] S109: Send a designated request to the target site.

[0108] The request header of the specified request at least includes XMLHttpRequest, and the parameters of the specified request at least include target=main and widget=XLite\View\Slidebar.

[0109] S110: Determine whether a navigation link can be extracted from the webpage content returned by the target site, wherein the navigation link has class="leaf"; if so, proceed to step S111. If not, proceed to step S112.

[0110] S111: Confirm that the target site type is an X-Cart website.

[0111] S112: directly access the homepage of the target site and determine whether each link on the homepage contains a specified parameter; wherein the specified parameter is route=product / category; if so, proceed to step S113; if not, proceed to step S114.

[0112] S113: Confirm that the target site type is an OpenCart website.

[0113] S114: Confirm that the target site is a non-SaaS site.

[0114] Among them, Shopify website, WooCommerce website, X-Cart website, and OpenCart website are all SaaS sites.

[0115] Specifically, first obtain the target site's website link (e.g., https: / / example.com) and append the product information's JSON file path, i.e., / products.json, to the end of the website link to obtain the target site's corresponding first website link (e.g., https: / / example.com / products.json). Determine whether the first website link can be opened normally. If the first website link can be opened normally, confirm that the target site is a Shopify website and extract its currency and language information. Specifically, search the webpage source code for the keyword Shopify.currency and extract the currency code (e.g., USD) enclosed in parentheses. Also, search the source code for Shopify.locale and extract the language code (e.g., en) enclosed in quotation marks.

[0116] If the first website link is confirmed to be unavailable, the WordPress REST API product data request path (i.e., / wp-json / wp / v2 / product?page=1&per_page=100) is appended to the end of the website link to obtain the second website link corresponding to the target site. The second website link is then checked to see if it can be opened. If so, the target site is confirmed to be a WooCommerce site. If the second website link is confirmed to be unavailable, a specified request is sent to the target site.

[0117] It should be noted that the request header must contain at least XMLHttpRequest (x-requested-with: XMLHttpRequest); the request parameters must contain at least target=main and widget=XLite\View\Slidebar. Determine whether navigation links can be extracted from the webpage content returned by the target site; navigation links must contain class="leaf". If navigation links can be extracted, the target site is confirmed to be an X-Cart website.

[0118] If it is confirmed that the navigation link cannot be extracted, the target site's homepage is directly accessed and the links on the homepage are checked to see if they contain the specified parameter, where the specified parameter is route=product / category. If the specified parameter is found to be present, the target site is confirmed to be an OpenCart site.

[0119] It should be noted that the Shopify website, WooCommerce website, X-Cart website and OpenCart website are all Software as a Service (SaaS) sites.

[0120] If the specified parameters are not included, the target site is confirmed to be a non-SaaS site. Further review the webpage's description (such as product details and title) and use translation tools to determine the language type. If no description is present, extract keywords from hidden data in the webpage's code (such as "language:en"). Also, scan the webpage's code for monetary amounts or the value following the keyword "currency" (such as "USD"). If all of the above steps fail, access the / robots.txt file, search for the sitemap.xml link, and directly crawl the content within the map.

[0121] Based on the above embodiment, in some embodiments, obtaining a link to a page in a corresponding site according to the type of each target site includes:

[0122] S115: When the target site is a SaaS site, access the specified API interface of the target site to obtain a link;

[0123] S116: When the target site is a non-SaaS site, access the website map file of the target site to determine whether the website map file is successfully accessed; if so, proceed to step S117; if not, proceed to step S118.

[0124] S117: Parse each uniform resource locator in the sitemap file to obtain a link.

[0125] S118: Crawl links in the target site using a depth-first search crawler algorithm, and / or load a page of the target site using a Web application automation testing tool and obtain links in the page.

[0126] Specifically, when the target site is a SaaS site, access the target site's designated API interface to obtain a link. For example, access the wp-json / wp / v2 / product interface to obtain detailed information links for all products.

[0127] When the target site is a non-SaaS site, access the target site's website map (sitemap.xml) file to determine whether the site map file has been successfully accessed. If so, parse the Uniform Resource Locators (URLs) in the site map file to obtain the links. If not, use the Depth-First Search (DFS) crawler algorithm, starting from the initial URL, to recursively access all links on the page until all links to product detail pages are captured, or the number of layers reaches a certain threshold, then the crawling stops. It should be noted that the initial URL needs to be manually specified by the technician, and the system automatically recursively crawls all links on the page. In addition, you can use web application automation testing tools (such as Selenium tools) to simulate a browser, load the target site's page, process dynamically loaded data, and save the source code and web page screenshots to obtain the links on the page.

[0128] In this way, we can automatically collect the initial URL and all product detail page links by adopting multiple strategies to adapt to e-commerce sites of different types and sizes.

[0129] In order to achieve clustering and classification of pages, based on the above embodiments, in some embodiments, the corresponding page is obtained according to each link, the XPath path of each tag in each page is extracted, and each XPath path is removed and classified and merged, including:

[0130] S121: Obtain the Hypertext Markup Language source code of the corresponding page according to each link.

[0131] S122: Obtain the XPath paths of all a tags in each hypertext markup language source code.

[0132] S123: Remove subscripts from each XPath path, classify and merge them.

[0133] Specifically, for each collected link, a Hypertext Markup Language (HTML) request is sent or the page content of the corresponding page is obtained through a rendering crawler, wherein the page content at least includes the HTML source code.

[0134] Further, get all the HTML source codes The XPath path of the tag is removed and the subscripts of each XPath path are removed and classified to generate a classification dictionary. It should be noted that the key of the classification dictionary is the XPath without the subscripts, and the value is a list of the original XPath. For example:

[0135] {

[0136] " / html / body / div / div / a" : [

[0137] " / html / body / div / div[1] / a" ,

[0138] " / html / body / div / div[2] / a" ,

[0139] " / html / body / div / div[3] / a"

[0140] ],

[0141] " / html / body / table / tr / td / a" : [

[0142] " / html / body / table / tr[1] / td[1] / a" ,

[0143] " / html / body / table / tr[1] / td[2] / a" ,

[0144] " / html / body / table / tr[1] / td[3] / a" ,

[0145] " / html / body / table / tr[2] / td[1] / a" ,

[0146] " / html / body / table / tr[2] / td[2] / a" ,

[0147] " / html / body / table / tr[2] / td[3] / a" ]

[0149] }

[0150] On this basis, in some embodiments, determining the common paths corresponding to the classification dictionaries and clustering the classification dictionaries according to the similarity of the common paths includes:

[0151] S124: Based on the slash character segmentation in each classification dictionary, determine the common path corresponding to each classification dictionary.

[0152] S125: Calculate the similarity of each common path according to the Levenshtein distance formula.

[0153] S126: Common paths with the same corresponding similarity are determined to be of the same type, and an initial clustering result is generated.

[0154] S127: Generate page clustering results according to the initial clustering results and the DBSCAN density clustering algorithm.

[0155] Specifically, according to the list of classification dictionaries and the segmentation of slash characters ( / ), the common path corresponding to each classification dictionary is determined, so that the classification dictionary can be converted into a list, as follows:

[0156] {

[0157] " / html / body / div" : 1 ,

[0158] " / html / body / table" : 1

[0159] }

[0160] [" / html / body / div"," / html / body / table"];

[0161] It's understandable that after the above steps, each page on each website becomes a classification dictionary. Finally, the similarity of each common path is calculated using the Levenshtein distance formula. The Levenshtein distance measures the degree of difference between two strings and is defined as the minimum number of editing operations (including character insertion, deletion, and substitution) required to transform one string into another. The formula is as follows:

[0162] ;

[0163] Among them, a and b are two strings, i and j represent the current character position; the value of cost is: if a i Not equal to b j , then it is 1, otherwise it is 0. For example, the Levenshtein distance between the strings "kitten" and "sitting" is 3, and the transformation process is as follows: kitten→sitten (replace the first character k with s), sitten→sittin (replace the fifth character e with i), sittin→sitting (insert the character g at the end).

[0164] After calculating the similarity of each public path according to the Levenshtein formula, public paths with the same similarity are classified as the same type to generate the initial clustering results. Finally, the page clustering results are generated based on the initial clustering results and the DBSCAN density clustering algorithm.

[0165] Correspondingly, each clustering result is classified, including:

[0166] S128: Classify the page clustering results according to artificial intelligence technology to obtain a details page set and a list page set.

[0167] The clustering results are classified using artificial intelligence (AI) technology to produce a set of detail pages and a set of list pages. It should be noted that a detail page set contains multiple detail pages, and a list page set contains multiple list pages. Each detail page typically corresponds to information about a single product. These pages provide detailed information about the product, including descriptions, specifications, prices, images, and user reviews. A detail page typically corresponds to a specific product. List pages display brief information about multiple products. These pages typically include product thumbnails, names, prices, and other information for users to browse and select. List pages display a collection of products, and users can select a product to view its detail page. Therefore, each product typically corresponds to a detail page, while a list page contains brief information about multiple products. The list page is a portal; users can click on a product link to access the corresponding detail page for more detailed information.

[0168] It should be noted that this embodiment does not limit the specific AI technology used. For example, it can be a large language model (LLM), a machine learning model, including traditional machine learning models such as decision trees, random forests or support vector machines (SVM), or a deep learning model, including convolutional neural networks or other deep learning architectures, which can be used for more complex feature extraction and classification tasks, especially when it is necessary to process large amounts of unstructured data.

[0169] In this embodiment, a clustering algorithm is used to cluster all links, and AI technology is used to sample and identify each category, thereby intelligently distinguishing between list pages and detail pages, and realizing the automation of page classification and identification in large-scale sites.

[0170] In order to extract target information, based on the above embodiment, in some embodiments, extracting target information from each target page includes:

[0171] S131: Determine whether there is structured data in the detail page source code in the detail page set; if so, proceed to step S132; if not, proceed to step S133.

[0172] S132: Extract target information from the detail page set through a structured data extraction function.

[0173] S133: Convert the page content in the detail page collection into Markdown format using the Hypertext Markup Language to Markdown technology.

[0174] S134: Identify target information in page content in Markdown format using artificial intelligence technology.

[0175] Specifically, it is determined whether the source code of the detail page in the detail page set contains structured data. If structured data exists, the target information in the detail page set is extracted using the structured data extraction function extruct.

[0176] It should be noted that extruct is a Python library for extracting structured data from web pages. Structured data is usually embedded in the source code of a web page in a specific format, making the data easier for machines to read and process. Common structured data formats include JSON-LD, Microdata, and RDFa. These formats are often used to embed semantic information about the content of the page in a web page so that search engines and other tools can better understand and display this data. Therefore, using extruct to extract this structured data can obtain detailed information about the content of the web page, usually including product name, price, description, product image URL, product ratings and reviews, and product availability and store information.

[0177] If there is no structured data, the page content in the detail page collection is converted into Markdown format using HTML to Markdown technology, and AI technology is further used to identify the target information in the page content in Markdown format, including but not limited to the product title, price, picture and other information.

[0178] In this embodiment, by combining AI technology and HTML to Markdown technology, it is possible to intelligently parse page content and identify information such as product titles, prices, and images.

[0179] To avoid the AI technology being unable to extract all target information, based on the above embodiment, in some embodiments, after identifying the target information in the page content in Markdown format using the artificial intelligence technology, the following steps are further included:

[0180] S135: Extracting relevant information corresponding to the target information in the simulated browser through context-aware visual attention technology.

[0181] The relevant information at least includes an XPath path.

[0182] S136: Traverse the detail page collection according to the XPath path to extract target information.

[0183] Specifically, if the AI model fails to accurately extract all information, context-aware visual attention (CoVA) technology is used to extract relevant information. This information includes at least the XPath path for the product title, price, and image. A simulated browser is used to take a screenshot of the page, and then COVA is used to extract the specific fields and locations. CoVA technology itself provides the ability to identify and locate web page elements, while XPath is the technology used to specifically identify the location of these elements. Combining the two enables efficient web page data extraction. Finally, the collection of detail pages is traversed according to the XPath path to extract the target information, such as the title, price, and image of all products.

[0184] In this embodiment, when AI technology cannot accurately identify, the Cova tool is used to extract the XPath path of the corresponding field and automatically generate parsing rules, which improves the adaptability of the system in large-scale sites and reduces maintenance costs and complexity.

[0185] In the above embodiments, the information extraction method is described in detail. This application also provides corresponding embodiments of the information extraction device.

[0186] Figure 2 This is a schematic diagram of an information extraction device provided in an embodiment of the present application. Figure 2 As shown, the device includes:

[0187] A first acquisition module 10 is configured to acquire a plurality of target sites and determine the type of each target site using a rule-based hierarchical feature detection technique; the rule includes at least a priority hierarchical rule and a feature fingerprint matching rule;

[0188] The second acquisition module 11 is used to acquire links to pages in corresponding sites according to the types of the target sites;

[0189] The first clustering module 12 is used to obtain the corresponding page according to each link, extract the XPath path of each tag in each page, and remove the subscripts, classify and merge each XPath path to generate a classification dictionary corresponding to each page;

[0190] The second clustering module 13 is used to determine the common paths corresponding to the classification dictionaries and cluster the classification dictionaries according to the similarity of the common paths to generate clustering results corresponding to each page;

[0191] The classification module 14 is used to classify each clustering result to obtain a target page containing target information;

[0192] The extraction module 15 is used to extract target information from each target page.

[0193] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, and they will not be repeated here.

[0194] Figure 3 This is a structural diagram of an information extraction device provided in an embodiment of the present application. Figure 3 As shown, the information extraction device includes:

[0195] Memory 20, for storing computer programs;

[0196] The processor 21 is configured to implement the steps of the information extraction method mentioned in the above embodiment when executing a computer program.

[0197] The information extraction device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0198] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented in at least one hardware form: a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is responsible for processing computing operations related to machine learning.

[0199] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the information extraction method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include but is not limited to data involved in the information extraction method.

[0200] In some embodiments, the information extraction device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0201] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the information extraction device, and may include more or fewer components than shown in the figure.

[0202] Finally, the present application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiment.

[0203] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0204] The above is a detailed introduction to an information extraction method, device, equipment and medium provided by the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of this application.

[0205] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. An information extraction method, characterized in that: include: Acquire multiple target sites and determine the type of each target site using a rule-based hierarchical feature detection technology; the rule includes at least a priority hierarchical rule and a feature fingerprint matching rule; Obtaining a link to a page in the corresponding site according to the type of each target site; Obtaining a corresponding page according to each link, extracting the XPath path of each tag in each page, and removing subscripts and classifying and merging each XPath path to generate a classification dictionary corresponding to each page; Determining a common path corresponding to each of the classification dictionaries, and clustering and grouping each of the classification dictionaries according to the similarity of each of the common paths to generate a clustering result corresponding to each of the pages; Classifying each of the clustering results to obtain a target page containing the target information; The target information in each target page is extracted.

2. The information extraction method according to claim 1, characterized in that Get multiple target sites, including: Determining the need to extract the target information; Determining the corresponding target sites according to the extraction requirements; The target sites are obtained through a site collection website or through a browser search.

3. The information extraction method according to claim 1, wherein: The type of each target site is determined using a rule-based hierarchical feature detection technique, including: Obtain the website link of the target site, and add the JSON file path of the product information to the end of the website link to obtain the first website link corresponding to the target site; Determining whether the first website link can be opened normally; If it is confirmed that the first website link can be opened normally, then confirm that the type of the target website is a Shopify website, and extract the currency information and language information of the target website; If it is confirmed that the first website link cannot be opened normally, then add the WordPress REST API product data request path to the end of the website link to obtain a second website link corresponding to the target site; Determining whether the second website link can be opened normally; If it is confirmed that the second website link can be opened normally, then the type of the target site is confirmed to be a WooCommerce website; If it is confirmed that the second website link cannot be opened normally, a specified request is sent to the target site; wherein the request header of the specified request at least includes XMLHttpRequest, and the parameters of the specified request at least include target=main and widget=XLite\View\Slidebar; Determine whether a navigation link can be extracted from the webpage content returned by the target site; wherein the navigation link has class="leaf"; If it is confirmed that the navigation link can be extracted, then the type of the target site is confirmed to be an X-Cart website; If it is confirmed that the navigation link cannot be extracted, directly access the homepage of the target site and determine whether each link on the homepage contains a specified parameter; wherein the specified parameter is route=product / category; If it is confirmed that the specified parameters are included, then the type of the target site is confirmed to be an OpenCart website; If it is confirmed that the specified parameter is not included, then the type of the target site is confirmed to be a non-SaaS site; Among them, the Shopify website, WooCommerce website, X-Cart website and OpenCart website are all SaaS sites.

4. The information extraction method according to claim 3, characterized in that Obtaining a link to a page in each target site according to the type of the target site includes: When the target site is a SaaS site, accessing a designated API interface of the target site to obtain the link; When the target site is a non-SaaS site, accessing a website map file of the target site to determine whether the website map file is successfully accessed; If so, parsing each uniform resource locator in the sitemap file to obtain the link; If not, the link in the target site is crawled by a depth-first search crawler algorithm, and / or the page of the target site is loaded by a Web application automation testing tool, and the link in the page is obtained.

5. The information extraction method according to any one of claims 1 to 4, characterized in that: Obtaining a corresponding page according to each link, extracting the XPath path of each tag in each page, and removing subscripts, classifying and merging each XPath path, including: Obtaining the Hypertext Markup Language source code of the corresponding page according to each link; Obtain the XPath paths of all a tags under each of the hypertext markup language source codes; Remove subscripts from each XPath path, classify and merge; Correspondingly, determining the common paths corresponding to the classification dictionaries and clustering the classification dictionaries according to the similarities of the common paths includes: Determining the common path corresponding to each classification dictionary based on the slash character segmentation in each classification dictionary; Calculating the similarity of each of the common paths according to the Levenshtein distance formula; Determine the common paths corresponding to the same similarity as the same type, and generate an initial clustering result; Generate page clustering results based on the initial clustering results and the DBSCAN density clustering algorithm; Correspondingly, each clustering result is classified, including: The page clustering results are classified according to artificial intelligence technology to obtain a details page set and a list page set.

6. The information extraction method according to claim 5, characterized in that Extracting the target information from each target page includes: Determine whether structured data exists in the detail page source code of the detail page set; If so, extract the target information from the detail page set using a structured data extraction function; If not, convert the page content in the detail page set into Markdown format using Hypertext Markup Language to Markdown technology; The target information in the page content in Markdown format is identified through artificial intelligence technology.

7. The information extraction method according to claim 6, characterized in that After identifying the target information in the page content in Markdown format using artificial intelligence technology, the method further includes: Extracting relevant information corresponding to the target information in a simulated browser by using context-aware visual attention technology; wherein the relevant information at least includes an XPath path; The detail page set is traversed according to the XPath path to extract the target information.

8. An information extraction device, characterized in that: include: A first acquisition module is configured to acquire a plurality of target sites and determine the type of each target site using a rule-based hierarchical feature detection technique; the rule includes at least a priority hierarchical rule and a feature fingerprint matching rule; A second acquisition module is used to acquire a link to a page in each target site according to the type of the target site; A first clustering module is configured to obtain a corresponding page according to each link, extract the XPath path of each tag in each page, and perform classification and merging on each XPath path without subscripts to generate a classification dictionary corresponding to each page; A second clustering module is used to determine the common paths corresponding to the classification dictionaries, and cluster the classification dictionaries according to the similarity of the common paths to generate clustering results corresponding to the pages; A classification module, configured to classify each of the clustering results to obtain a target page containing the target information; The extraction module is used to extract the target information from each target page.

9. An information extraction device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the information extraction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the information extraction method according to any one of claims 1 to 7.

Images