Advanced response handling in web data collection

Abstract

Advanced response processing in web data collection discloses processor implemented apparatuses, methods, and systems for processing unstructured raw HTML responses collected in the context of a data collection service, in one embodiment, the method includes receiving raw unstructured HTML documents and extracting textual data with associated meta information that can include style and formatting information. In some embodiments, data field labels and values can be assigned to the extracted text blocks, which are classified based on processing of machine learning algorithms. Further, the extracted data blocks can be grouped and regrouped together and presented as a single data point. In another embodiment, the system can aggregate and present the textual data with associated meta information in a structured format. In certain embodiments, the machine learning model can be a model trained on a pre-created training dataset that is labeled manually or in an automated fashion.

Description

Technical Field

[0001] The methods and systems detailed in this paper involve processing response data in the context of a data collection service, where the processing employs a set of parsing tools assisted by a trained machine learning classification model. Background Technology

[0002] Web scraping (also known as screen scraping, data mining, or web gathering) is, in its most general sense, the automated collection of data from the internet. Technically, it is the practice of collecting data from the internet by any means other than humans using a web browser or a program interacting with an application programming interface (API). Web scraping is typically accomplished by executing a program that queries a web server and automatically requests data, then parses the data to extract the requested information.

[0003] Web crawlers—programs written for web crawling—have significant advantages over other ways of accessing information, such as web browsers. While web browsers are designed to present information in a human-readable way, web crawlers excel at rapidly collecting and processing large amounts of data. Unlike opening one page at a time through a monitor (as web browsers do), web crawlers can collect, process, aggregate, and render large databases of thousands or even millions of pages at once.

[0004] Sometimes, websites allow another automated way to transfer their structured data from one program to another via an API. Typically, the program requests some type of data from the API via the Hypertext Transfer Protocol (HTTP), and the API returns this data from the website in a structured form. It acts as an intermediary for data transfer. However, using an API is not considered web crawling because the API is provided by the website (or a third party), and it eliminates the need for a web crawler.

[0005] APIs can transfer well-formatted data from one program to another, and using them is easier than building a web crawler to retrieve the same data. However, APIs are not always available with the desired data. Furthermore, APIs often use capacity and rate limits and restrict the type and format of data. Therefore, users may attempt to use web crawling with data that does not exist or is restricted in any way by the API.

[0006] Typically, web crawling involves the following steps: retrieving Hypertext Markup Language (HTML) data from a website; parsing the data for desired target information; saving the desired target information; and repeating the process on another page if necessary. A web crawler is a program designed to perform all these steps on a large scale. A related program—a web crawler (also known as a web spider)—is a program or automated script that performs the first task, namely, navigating the web in an automated manner to retrieve the raw HTML data of the websites visited (a process also known as indexing).

[0007] Fetching activities can be performed by various types of fetching applications, which can generally be categorized as browsers, headless browsers, command-line tools, programming language libraries, etc.

[0008] A browser is an application that runs within a computing device, typically within the context of an end-user session, and is functional enough to accept user requests, pass them to a target web server, process responses from the web server, and present the results to the user. Browsers are considered, for example, tools that enable user-side scripting, such as those capable of executing and interpreting JavaScript code.

[0009] Headless browsers—web browsers without a graphical user interface (GUI). Headless browsers provide automated control over web pages in an environment similar to popular web browsers, but executed via a command-line interface or using network communication. They are particularly useful for testing web pages because they can render and understand HTML like a browser, including style elements such as page layout, colors, font selection, and the execution of JavaScript and AJAX, which are typically unavailable when using other testing methods. Two main use cases can be identified:

[0010] Scripted web page testing aims to identify errors, and it is necessary to closely mimic user activity.

[0011] Web crawling—must resemble user activity to avoid blocking; that is, the request should have all the attributes of an organic web browsing request.

[0012] Headless browsers are considered to be user-side scripting-enabled tools, for example, capable of executing and interpreting JavaScript code.

[0013] Command-line tools—GUI-less applications that allow generating and submitting web requests via a command-line terminal such as CURL. Some tools in this category may have a GUI wrapped around them, but graphical elements will not overlay the display of the results of the HTTP requests. Command-line tools have limited functionality because they cannot execute and interpret JavaScript code.

[0014] Programming language libraries are collections of behavior implementations written in the language, with well-defined interfaces for invoking those behaviors. For example, when a scraping application needs to invoke a specific HTTP method to perform a scraping request, it can use a library containing that method to make the system call, instead of implementing those system calls repeatedly within its own code. Furthermore, they provide behaviors for reuse by multiple independent programs that invoke the behaviors provided by the library via language mechanisms. Therefore, the value of a library lies in the reusability of its behaviors. When a program calls a library, it obtains the behavior implemented within that library without having to implement it itself. Libraries encourage modular code sharing and simplify code distribution. Programming language libraries have limited functionality because they cannot execute or interpret JavaScript code unless a library exists as a wrapper for another tool capable of executing client-side scripts.

[0015] The previous combinations of basic proxy types implemented HTTP protocol methods and client-side scripting to varying degrees.

[0016] The types of web crawling applications listed above differ in their technical capabilities, typically due to their different development purposes. When sending an initial request to the target web server, all listed types of crawling applications pass the parameters necessary to submit and process the web request. These include HTTP parameters—headers, cookies, the version of the HTTP protocol they support and intend to communicate with—and TCP parameters exposed when initiating the underlying Transmission Control Protocol (TCP) session for the HTTP request (e.g., TCP window size and other parameters). As mentioned above, browsers and headless browsers can handle JavaScript files obtained within the web server's response, for example, submitting configuration settings via JavaScript at the time of the request, something command-line utilities cannot do.

[0017] When processing web server responses, all listed types of crawling applications are capable of obtaining, interpreting, rendering, or otherwise processing and presenting HTTP metadata and the main HTML document. However, some listed crawling applications lack the ability to process additional files obtained from the web target's response, such as executing scripted code on the client side. Therefore, the actual classification of web harvesting tools is based on their ability to execute and interpret JavaScript code.

[0018] Further disclosure of the entire data collection process could focus on outlining the structure of a standard web server request.

[0019] The response obtained from a web server typically includes the following parts:

[0020] - HTTP metadata, including HTTP headers, cookies, and HTTP response codes;

[0021] -The main HTML document;

[0022] - Additional files required for processing and rendering the final version of a webpage: images, Cascading Style Sheets (CSS) files, and JavaScript (JS) scripts.

[0023] A simple HTML file contains data formatted with baseline HTML code, while an MHTML file is a text file containing complete response data: the main document (HTML), a .css file—information about the styles of each element, images, and a JavaScript file containing uncompiled script code to be executed to render the final webpage.

[0024] The Document Object Model (DOM) is a programming interface for HTML and XML documents. It represents a webpage, allowing programs to modify its structure, style, and content. The DOM is an object-oriented representation of a webpage, ensuring that programming languages ​​can connect to the page and manipulate its elements. Most modern browsers implement the W3C DOM and WHATWG DOM standards. To further extend this, all properties, methods, and events that can be used to manipulate and create webpages are organized into objects, such as the entire document, headings, tables within the document, table headings, and text within table cells.

[0025] The modern DOM is built using multiple APIs that work together. The core DOM defines objects that fundamentally describe documents and the objects within them. The core DOM is extended by other APIs as needed, which add new features and capabilities to the DOM. For example, the HTML DOM API adds support for representing HTML documents to the core DOM.

[0026] XPath is a fundamental element for manipulating web pages, enabling navigation across the DOM hierarchy. The XPath language is based on a tree representation of an XML document and provides the ability to navigate within the tree and select nodes according to various criteria. In popular usage (though not in the official specification), XPath expressions are often simply referred to as "XPath," where it contains the location of any element on a web page using the HTML DOM structure, defined by syntax or language for finding any element on the page using XML path expressions.

[0027] Since manually processing large amounts of data is rarely efficient or even infeasible, supporting methods have been involved in automating data analysis. One such method is machine learning.

[0028] Machine learning can be broadly defined as a computational approach that uses aggregated data to improve performance or make accurate predictions. Here, aggregated data refers to past information available to machine learning algorithms, typically in the form of electronic data that has been collected and made available for analysis. Summary of the Invention

[0029] A major drawback when processing requests or request streams of specific data points is dealing with irrelevant or redundant portions of the collected data. For example, companies collecting data from e-commerce websites and their product pages typically encounter several problems with standard-based parsers, the most significant being that each website adheres to a unique, custom layout. Mitigating this by developing custom parsing and data collection tools relies heavily on the ongoing development and maintenance of parsing code—for example, the code used for pricing and product intelligence information. Furthermore, the layout of target web pages can change frequently, requiring additional effort for data extraction and selective adjustments. The fact that web servers present multiple versions of the same web page to clients (based on the country or browser language settings of the request origin) can exacerbate this problem. This again forces data collection companies to adapt their already built parsers to different localized product page layouts.

[0030] Building a parser for each individual e-commerce website is extremely time-consuming and resource-intensive. Each potential target website has a unique HTML structure and requires unique rules to extract relevant elements, such as prices, titles, or descriptions. As mentioned earlier, parsers must be constantly maintained and updated because websites can change their layout and HTML structure at any time, making previous parser rules obsolete. Machine learning-enabled platforms for data extraction and parsing can automatically adapt to such changes and be trained to parse relevant elements without requiring a rule set for each client and are unaffected by changes in website layout.

[0031] To successfully collect public data and process the results with minimal errors and inconsistencies, ensuring consistency with the original data collection requirements, crawling applications need to adopt an adaptive approach. One exemplary method to ensure content is properly processed, categorized, and presented is advanced web page data transformation and classification based on a machine learning classifier model. This model is used to parse and inspect responses received from the target platform, classify the collected data according to the requested data model, and feed the correctly labeled data back to the crawling platform. Input for classification can be obtained by extracting the HTTP portion of the response, crawling the extracted HTML data into text blocks containing the exact information expected by the requester while stripping away irrelevant information, and normalizing and formatting the data accordingly. However, in some embodiments, the non-HTML portion of the response can be part of the classification input, such as style elements assigned to objects within the web page.

[0032] Categorized outputs can be fed into the final response data aggregation and presented to the requester. Among other benefits, these methods increase the possibility of correctly tagging and categorizing request-customized content at the crawling platform for further presentation to the requesting client.

[0033] On the one hand, the embodiments detailed herein disclose a method and system for examining content returned by a content provider by processing text elements extracted from collected HTML pages, parsing data, identifying appropriate categories of the parsed elements, assigning categories accordingly, and feeding the categorized data back to the crawling platform, wherein the response processing is based on a trained machine learning classification model. Attached Figure Description

[0034] The features and advantages of the exemplary embodiments described herein will become clear to those skilled in the art upon reading the following description and referring to the accompanying drawings.

[0035] Figure 1 This is an exemplary component diagram that illustrates the overall architecture of the components and the context of at least one aspect of the solution's functionality.

[0036] Figure 2 It is a more detailed description of the data extractor platform within the entire service provider infrastructure architecture.

[0037] Figure 3 The construction of a training dataset for a data point classifier model is demonstrated.

[0038] Figure 4 The lifecycle and overall operation of the data point classifier model are described, starting from the initial training phase, handling of actual requests, and the recurrent feedback model that updates the training dataset with classification decisions that have been manually checked.

[0039] Figure 5 This is a demonstration of the overall process of data collection requests.

[0040] Figure 6a depicts the classification and transformation of the collected data.

[0041] Figure 6b is a continuation of Figure 6a, further illustrating the process of data classification and transformation.

[0042] Figure 7 It is an exemplary computing system that performs the disclosed methods. Detailed Implementation

[0043] Descriptions of some common terms may be helpful and are included herein for convenience, and are intended to be interpreted in the broadest possible sense. Elements not explicitly defined in the specification shall have the meaning as would be understood by one of skill in the art. Elements 104, 106, 108, and 210 identify portions of the service provider's infrastructure, while elements 102, 130, 132, 134, 136, and 140 depict external components or systems.

[0044] User device 102 can be any suitable user computing device, including but not limited to smartphones, tablets, personal computing devices, laptops, gaming devices, vehicle infotainment systems, smart appliances (e.g., smart refrigerators or smart TVs), cloud servers, mainframes, laptops, desktops, workstations, mobile devices, or any other electronic devices used to make a grab request.

[0045] Service Provider Infrastructure 104 (SPI 104) is a combination of elements including a platform that provides services for collecting data from the Internet by executing user-submitted data collection requests, processing the collected data, and handing the data over to the requesting user.

[0046] The crawling agent 106 is a component of the service provider infrastructure 104, specifically responsible for containing and running the crawling application, which executes crawling requests originating from business users and accepts such requests from users. Therefore, another function of this element is to perform data collection operations based on the requests submitted to it. After obtaining response data from one or more target systems, the crawling agent 106 either returns the data to the requester or, if necessary, performs additional processing on the collected data.

[0047] One aspect of the Capture Agent 106 function is that, after receiving a response from the target, it submits it to the component responsible for additional data evaluation, classification, and transformation operations for further processing.

[0048] The Universal Data Extractor (UDE) 210, a component of SPI 104, is responsible for accepting calls from crawling agent 106 and evaluating data submitted within the call, where the data is content acquired during one or more data collection requests. The evaluation of the data includes preprocessing the data contained therein, extracting relevant data points consistent with the original data collection requests, classifying and labeling the resulting content, and finally returning the classified and labeled data to crawling agent 106, thereby providing the probability percentiles of the identified classifications. UDE 210 includes several components that provide the described functionality.

[0049] Application Programming Interface (API) 211 is an internal component of UDE 210, responsible for external communication, integration, and internal communication between UDE 210 components.

[0050] Application Programming Interface (API) 211 is performing classification and labeling of data provided by crawling agent 106. The classification employs data point classifier model 214, which is trained on a dataset specifically constructed based on multiple data collection responses previously collected and labeled.

[0051] Rendering Engine 212 is an internal component of UDE 210 that performs the rendering of the data to be categorized. It can perform additional tasks, such as separating different HTML content from the MHTML data submitted for analysis and categorization. Rendering can be performed using toolsets such as headless browsers, as well as other options.

[0052] HTML parser 213 is an internal component of UDE 210 that extracts text information from HTML data isolated during the rendering of MHTML content.

[0053] The Data Point Classifier Model (DCM) 214 is an internal component of the UDE 210 that classifies new data points within the provided data based on patterns observed from previous data (i.e., the training dataset).

[0054] Practical machine learning-based classification models can be bag-of-words, Naive Bayes, support vector machines, logistic regression, random forest classifiers, extreme gradient boosting models, convolutional neural networks, or recurrent neural networks.

[0055] The Dataset Preparation Unit (DPU) 215 is a container object that includes all the components and functionality required to preprocess data before it is submitted for classification. The toolset contained therein is described in an exemplary manner in the current embodiment and can be extended with additional tools adapted to the input requirements of the data point classifier model 214.

[0056] Proxy 130 and 132 indicate a large number of exemplary proxy servers (computer systems or applications) opened for client connections, acting as intermediaries for requests from clients seeking resources from other servers. The client connects to the proxy server, requesting services such as files, links, web pages, or other resources available from different servers. The proxy server evaluates the content request and forwards the request to one or more target resources containing the content. After obtaining the content, the proxy server typically forwards it to the original requester, but the proxy may also perform other actions (e.g., returning error information). On the one hand, in at least one embodiment detailed herein, the proxy server may not fully understand the actual content obtained for the original requester; for example, in the case of an encrypted HTTPS session, if the proxy is not the decryption endpoint, then the proxy acts as an intermediary blindly forwarding data without knowing what is being forwarded. However, the metadata of the response is always visible to the service provider, such as HTTP headers. This functionality is necessary for the proxy to correctly forward the obtained data to the correct requester (end user or intermediary proxy device). Agents 130 and 132 are presented herein as a simple indication that more than one proxy server may be maintained at the service provider infrastructure 104 or that more than one proxy server may be externally available for performing data collection operations. The embodiments should not be limited to proxies belonging to a service provider. Proxies may be owned and managed by a third party; however, it is assumed that the service provider infrastructure 104 has access and can use such proxies to service scraping requests.

[0057] Objectives 134 and 136 indicate a large number of exemplary web servers serving content accessible via HTTP / HTTPS protocols. Objectives 134 and 136 are presented herein as simple indications that there may be more than one objective, but should not be construed in any way as limiting the scope of this disclosure. There can be an unlimited number of objectives on a network.

[0058] Network 140 is a digital telecommunications network that allows nodes to share and access resources. Examples of networks include: Local Area Network (LAN), Wide Area Network (WAN), Campus Network (CAN), Metropolitan Area Network (MAN), Home Area Network (HAN), Intranet, Extranet, Internet, and the Internet.

[0059] Several exemplary methods for constructing, training, and using data point classifier models to classify response data include Figure 1The system depicted herein illustrates an exemplary component diagram within the context of the overall architecture of the disclosed components and at least one aspect of the solution's functionality. In at least one aspect, this embodiment illustrates communication between user equipment 102 and service provider infrastructure 104 to reach any target 134, 136 on network 140. Service provider infrastructure 104 includes a crawling agent 106 and a general data extractor 210.

[0060] Figure 1 An embodiment is demonstrated in which user equipment 102 and proxy servers 130, 132 are not part of service provider infrastructure 104. User equipment 102 can be any device that can be connected to service provider infrastructure 104 via a network, in some cases via the Internet. Such a device can be a computer, mobile phone, tablet, or any other electronic device. Figure 1 Targets 134 and 136, which are approached via agents 130 and 132, are located outside the service provider infrastructure 104 and are communicatively attached to any network 140. In some embodiments, network 140 is the Internet.

[0061] Figure 1 The elements within the service provider infrastructure 104 are further demonstrated, starting with a crawling agent 106 that accepts and processes requests from user equipment 102. In executing the request and responding to a response obtained from the target, the crawling agent 106 communicates with a general data extractor 210, submitting the obtained response for analysis and ultimately classification. The resulting data presents multiple classified and labeled data points, which in turn constitute a dataset suitable for return to user equipment 102. Coupled with the classification labels are probability percentiles used for classification.

[0062] Figure 2 This is a more detailed description of the general data extractor 210. The components and functions included therein are employed in two operational flows: -1) training of the data point classifier model and 2) processing of the routine data collection response to extract and classify the relevant data points (in...) Figure 4 (As described in the text).

[0063] The general data extractor 210, a component of SPI 104, is responsible for accepting calls from crawling agent 106 at API 211 and evaluating the data submitted within the call, where the data is content obtained during a data collection request or multiple requests. The data evaluation includes preparing the data contained therein by rendering the input data using rendering engine 212, extracting relevant text information according to the original data collection request using HTML parser 213, processing the extracted text blocks and corresponding metadata by dataset preparation unit 215 (DPU 215) for deriving classification attributes, classifying and labeling the resulting data points based on the processing using data point classifier model 214 (DCM 214), and finally returning the classified and labeled data to crawling agent 106 to provide probability percentiles for the identified classification.

[0064] The process of classifying and labeling data points can be operated on any voluntary set of categories. In the exemplary embodiment of this application, an e-commerce webpage has been selected as the focus area. The following is an exemplary list of possible categories: price, previous price, title, item description, item ID, brand name, availability, additional information, images, etc.

[0065] There are at least two possible methods to parse multiple categories:

[0066] - Train a single model to solve multi-class problems. In this case, the dataset preparation unit 215 uses only one model that can return a prediction of the class corresponding to the text block and a probability score for each class. The main advantage of this method is that a single model processes the data only once. However, the accuracy of the results provided is relatively low.

[0067] - Train a separate model for each category. This is a more accurate approach, but it requires multiple models to classify the data over repeated cycles, once per category. Improved accuracy is ensured by customizing each model based on the specific latent attributes and parameters of each category.

[0068] The process of training the data point classifier model 214 requires an initial training dataset containing a large amount of HTML data. Each HTML data point should be manually labeled according to the training process run on the dataset.

[0069] Figure 3 The process of constructing the training dataset 300 is described, wherein an initial set 311 of MHTML data aggregated from the results of multiple instances of data collection 310 is submitted to HTML extraction 320 by rendering MHTML, which results in the extraction of HTML data 321.

[0070] The resulting data is then submitted to the Data Preparation 330 process, which includes the following steps:

[0071] - Parse the HTML portion of the response at step 331;

[0072] - At step 332, extract text blocks from the parsed HTML along with the xpath parameter of each text block, and optionally concatenate text blocks representing the same information item;

[0073] - Data cleaning, including removing all irrelevant text blocks at step 333;

[0074] - In step 334, extract the metadata associated with each text block, including HTML tag / class attributes, text attributes, style attributes, and XPath attributes. These attributes can ultimately be defined by custom numerical values ​​and combined into the string.

[0075] - In step 335, the identified text blocks are combined with the associated metadata to establish the data points to be classified.

[0076] During data point labeling 340, data points are labeled at step 341 to ensure correct input when constructing the training dataset 351 during dataset construction 350. The purpose of manual labeling is to ensure that the input used to train the data point classifier model 214 contains data that promotes correct prediction behavior, thus ensuring better classification accuracy. The processed dataset construction phase 350 results in the preparation of the training dataset 351.

[0077] Figure 4 Demonstrates a complete data point classifier model 214 lifecycle 400, starting from model training 410, where data comes from... Figure 3 The training dataset 351, constructed from the training dataset 300, is presented to the untrained model 412. After training, the model reaches its production stage (data point classifier model 214) in the new data processing stage 430, ready to handle actual requests to classify new data for classification 452. The classification result—classification decision 431—is submitted back to the data collection process 450, where classification processing 453 is performed, and the result is handed over to the crawling session 451. Finally, the response data 461 is submitted to the customer during the customer handover stage 460.

[0078] In another aspect of the embodiments presented herein, the adaptability percentage of the classification decision 431 instances constructed during the new data processing 430 phase can be integrated into the training dataset 351, assuming that the analyzed data and the resulting classifications undergo a model training set enhancement process 420, where their correctness is confirmed during quality assurance 421 and they are integrated into the model training dataset 351. Continuous quality assurance inputs used to update the training dataset 351 ensure the correctness of future classifications by the data point classifier model 214.

[0079] Figure 5 This is a sequence diagram outlining the route taken by a crawl request. The process begins with user device 102 submitting a crawl request at step 502, where crawl proxy 106 acts as the entry point for service provider infrastructure 104, and processing of the request at step 504 includes further execution via proxy 130. Proxy server 130 then submits the request to target 134 (step 506). At step 508, target 134 processes the request and delivers the requested content to proxy server 130 at step 510, which is then delivered by proxy server 130 to crawl proxy 106 at step 512. At step 514, response data is submitted to universal data extractor 210 for classification and labeling, where at step 516 the response is processed, analyzed, and classified within UDE 210, and at step 518 it is returned to crawl proxy 106 in the form of a structured and categorized dataset, which is finally delivered to the original requester either directly or via an asynchronous callback request at step 520.

[0080] Figure 6A and 6B It describes in more detail the route taken by the response data and the operations the data undergoes in order to transform the raw, unprocessed, and unstructured MHTML format obtained from the target web server into a structured, categorized, and labeled dataset.

[0081] Starting within the crawling agent 106, at step 602, the response obtained from the target web server is submitted as a whole for classification and transformation to the general data extractor 210, i.e., the integration interface of UDE 210—API 211. The data here is an MHTML file, a text file containing complete response data: the main document (HTML), CSS files, images, JavaScript files, etc. Therefore, at step 604, the data is passed to the internal UDE 210 component—the rendering engine 212, which in this embodiment is a headless browser—which then separates the HTML portion of the submitted data at step 606 and returns the result to API 211. Here, at step 608, the HTML file is handed over to the parser 213 component of UDE 210 for extracting text from the HTML input using the corresponding XPath elements at step 610. Assuming that the text blocks describe relevant information units, then at step 612, the parser 213 may have to combine text elements from different HTML elements using a text block concatenation algorithm. At step 614, parser 213 returns the output as a text block with a corresponding xpath to API 211, where the xpath is essentially a path to the HTML element from which the text block is extracted.

[0082] During the next step 616, API 211 obtains meta-information for the identified text block by submitting a request to rendering engine 212, wherein the request contains the xpath for the expected text block, and at step 618, rendering engine 212 extracts and returns the requested tags, classes, ids, variables, and style elements corresponding to the text block.

[0083] At this stage, API 211 has the original HTML file, the text blocks extracted from it, and the XPath, HTML tags / classes, and style elements associated with the text blocks, and prepares the data for processing at step 620.

[0084] Figure 6b is a continuation of Figure 6a, further illustrating how API 211 continues to submit data for classification to data preparation unit 215 at step 622. Data preparation unit 215 cleans the data at step 624, removing redundant elements for classification, and performs steps to identify and evaluate attributes relevant to the classification of each text block. Since the ultimate goal is to identify which text block belongs to which category (e.g., price, old price, title, description, etc.), an identification step is performed for each text block at step 626. A more detailed description of data cleaning and preparation is disclosed as one of many potential ways to prepare data for classification based on machine learning models.

[0085] During data cleaning at step 624, some text blocks are removed from the dataset. The fewer data points in the dataset, the easier it is for the model to find the correct text block for the desired category. For example, if the text block style attribute "visibility" equals "hidden," then the text block is considered irrelevant because it is not visible in the HTML file and is likely either to contain no relevant information or to be reliably assigned to any category, and in both cases should be ignored.

[0086] A block of text containing no text or only a single character is another example of redundant data. In this case, it either does not carry valuable information or cannot be reliably assigned to any category.

[0087] Step 626 includes activities performed to identify meta-attributes associated with each text block and derive them from meta-information, including:

[0088] HTML Attributes: In this step, the data preparation unit extracts information from HTML tags, classes, IDs, and variables. Potential attributes include:

[0089] - A text block element contains certain tags from the HTML source;

[0090] - Text block elements contain certain classes from the HTML source;

[0091] - Text block elements contain certain variables from the HTML source.

[0092] Text attributes: In this step, the data preparation unit extracts information from the extracted text elements. Here are some examples of text-related attributes:

[0093] - The character count of the text block;

[0094] - Word count of the text block;

[0095] - Sentence count of the text block;

[0096] - The text block contains specific keywords;

[0097] - The proportion of numbers in the text block;

[0098] - The proportion of special characters in the text block;

[0099] - A text block contains dots.

[0100] Style attributes: Basic and relevant information can be included within style-related attributes, as each text block can have over 160 style attributes containing information about the text block that can provide valuable input, helping the data point classification model 314 predict the text block category. Some examples of important style attributes are:

[0101] Position of text blocks in HTML;

[0102] Text block color;

[0103] Text block font styles;

[0104] Text block font size;

[0105] Whether the text block is underlined.

[0106] XPath related properties: Several properties can be derived from the XPath parameters associated with the text block:

[0107] The depth of text block elements in HTML;

[0108] Does a specific keyword exist in the XPath of a specific text block?

[0109] At step 628, the results of attribute identification and evaluation are returned to API 211, where, at step 630, the entire data point is submitted to the data point classifier model 214. At this point, the data point contains a single text block and a classification attribute identified, evaluated, and prepared by the dataset preparation unit 215. Actual classification occurs at step 632, where the model classifies and predicts the category for each data point. At step 634, the model returns the classification for each data point, along with the probability scores associated with the classification and the data point, to API 211. At this stage, the dataset at API 211 contains each data point (text block) predictively associated with a specific category. For example, if the data preparation unit 215 and the data point classifier model 214 predict which element is the price, then API 211 can at this point select only the text element with the highest price probability score and consider it as the "price" for further analysis steps.

[0110] In some embodiments, the general data extractor 210 may operate based on multiple classification models (a set of categories), wherein the requesting user device may submit a preference for which classification model to request via parameters of the request.

[0111] In another embodiment, the classification model used may be an implementation of one of the following machine learning models: bag-of-words, Naive Bayes algorithm, support vector machine, logistic regression, random forest classifier, extreme gradient boosting model, convolutional neural network, or recurrent neural network.

[0112] In yet another embodiment, the classification decision is submitted to the classification platform for quality assurance, whereby the assigned classification is checked and verified. The quality-assured classification decision is deemed correct and becomes part of the training of future machine learning classification models, and is incorporated into the corresponding training set.

[0113] Any of the embodiments described above herein can be rearranged and / or combined with other embodiments. Therefore, the concepts herein are not limited to any of the embodiments disclosed herein. Furthermore, embodiments may be entirely in hardware or may include both hardware and software elements. Some embodiments may be implemented in software, including but not limited to firmware, resident software, microcode, etc. Figure 7 A computing system 700 is illustrated, wherein a computer-readable medium 706 can provide instructions for performing any of the methods disclosed herein.

[0114] Furthermore, embodiments may take the form of a computer program product accessible from a computer-readable medium 706, which provides program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, the computer-readable medium 706 may be any means capable of tangibly storing a program for use by or in connection with an instruction execution system, apparatus, or device, including computer system 700.

[0115] Media 706 can be any tangible electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device). Examples of computer-readable media 706 include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), NAND flash memory, read-only memory (ROM), hard disk, and optical disc. Some examples of optical discs include compact disc-read-only memory (CD-ROM), compact disc-read / write (CD-R / W), and digital versatile disc (DVD).

[0116] A computing system 700 suitable for storing and / or executing program code may include one or more processors 702, which are directly or indirectly coupled to memory 708 via a system bus 710. Memory 708 may include local memory used during the actual execution of the program code, mass storage devices, and cache memory that provides temporary storage for at least some program code to reduce the number of times code is retrieved from mass storage devices during execution. Input / output or I / O devices 704 (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system directly or via an intermediate I / O controller. Network adapters may also be coupled to the system to enable the computing system 700 to be coupled to other data processing systems, such as via a host system interface 712, or to remote printers or storage devices via an intermediate private or public network. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0117] Identifiers such as “(a)”, “(b)”, “(i)”, and “(ii)” are sometimes used for different elements or steps. These identifiers are used for clarity and do not necessarily specify the order of elements or steps.

[0118] While several embodiments have been described, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the embodiments detailed herein. Therefore, the specification and figures are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this teaching. Benefits, advantages, solutions to problems, and any one or more elements that may cause any benefit, advantage, or solution to occur or become more apparent should not be construed as key, essential, or necessary features or elements of any or all claims. The invention is defined solely by the appended claims, including any modifications made during the pending period of this application and all equivalents of those published claims.

[0119] Furthermore, in this document, relational terms such as "first" and "second" may be used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes, has, contains, or contains a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Elements preceding "comprises…a," "has…a," "contains…a," or "includes…a" do not exclude the presence of additional identical elements in processes, methods, articles, and / or apparatuses that include, have, contain, and / or contain that element, unless otherwise expressly stated herein. Unless otherwise expressly stated herein, the terms "a" and "an" are defined as one or more. The terms “about,” “approximately,” or any other form thereof are defined as close to, as understood by one of ordinary skill in the art. A device or structure “configured” in a certain way is configured at least in that way, but may also be configured in ways not listed. Indications for elements may be in singular or plural forms, but this does not limit the scope of this disclosure, and the same teachings may apply to multiple objects, even if the object is mentioned in its singular form in the present application.

[0120] The embodiments detailed herein are provided to allow the reader to quickly determine the nature of the technical disclosure. It is understood that this submission is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed description, for the purposes of this disclosure, multiple features are demonstrated to be combined in various embodiments. This method of disclosure should not be construed as reflecting an intention that the claimed embodiments require more features than expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter lies in having fewer features than in a single disclosed embodiment in at least some instances. Therefore, the following claims are hereby incorporated into the detailed description, each claim being an independent, separately claimed subject matter.

[0121] This disclosure proposes a method for classifying text blocks in a data collection response, including:

[0122] (a) Receive a data collection response captured from a data collection target in accordance with a data collection request, wherein the request originates from the user equipment that issued the request;

[0123] (b) Obtain multiple text blocks from the data collection response;

[0124] For each of the plurality of text blocks:

[0125] (c) Obtain the path to the location of the corresponding text block within the data collection response;

[0126] (d) Collect meta-attributes within the data collection response that describe the display and functional characteristics of text blocks within the page specified by the data collection response via a path;

[0127] (e) Derive category attributes from text blocks, paths, and meta attributes;

[0128] (f) Perform a trained machine learning classification model on the categorical attribute to determine the classification of the text block;

[0129] (g) Construct a dataset using the plurality of text blocks and the corresponding classification determined in (f) for each of the plurality of text blocks; and

[0130] (h) Transmit the dataset to the user device that issued the request.

[0131] A method is proposed in which the received data collection response is in HTML format.

[0132] A method is proposed in which the received data collection response is in MHTML format.

[0133] The proposed method also includes rendering MHTML to extract an HTML file, wherein obtaining (b) includes obtaining the plurality of text blocks from the HTML file.

[0134] A method is proposed, wherein collection (d) includes collecting meta-attributes from the HTML and non-HTML portions of the data collection response.

[0135] This method is proposed, where data points include text blocks within the data collection response, associated meta attributes, corresponding paths, and source HTML elements.

[0136] The proposed method also includes preprocessing data points to derive categorical attributes from associated meta-attributes, and assigning the associated categorical attributes to corresponding text blocks.

[0137] A method is proposed in which the classification attribute assigned to a text block is derived from at least one of the following: HTML tags, classes, IDs and groups of variables of the text block, text attributes of the text block, style attributes of the text block, and path.

[0138] A method is proposed in which the identified categorical attributes are registered and assigned to text blocks in numerical format.

[0139] A method is proposed, in which meta-attributes include style information.

[0140] The proposed method also includes concatenating individual text blocks from the plurality of text blocks.

[0141] A method is proposed, wherein execution (f) includes applying a classification attribute to multiple machine learning classification models, each of which is trained to identify whether a text block belongs to a category.

[0142] A method is proposed in which the plurality of machine learning classification models each determine a classification probability, the classification probability indicating the likelihood that a text block belongs to the category that the corresponding machine learning classification model was trained to detect.

[0143] A method is proposed in which the trained machine learning classification model used is at least one of the following, but not limited to: bag of words, Naive Bayes algorithm, support vector machine, logistic regression, random forest classifier or extreme gradient boosting model.

[0144] A method is proposed in which classification decisions are submitted to a classification platform for quality assurance, wherein the assigned classifications are checked and validated through crowdsourcing.

[0145] A method is proposed in which quality-assured classification decisions are classified as correct and become part of the training of future machine learning classification models and are incorporated into the corresponding training set.

[0146] A method is proposed in which the data collection response includes non-textual information.

[0147] A method is proposed in which a trained machine learning classification model is used to process content delivered within non-textual information.

[0148] A method is proposed in which the delivery (h) is performed via an intermediary component such as a fetching agent.

[0149] A method is proposed in which if (b) does not return a recognizable text block, then a data collection response is not submitted for classification, and the request is resubmitted as a data collection request.

Claims

1. A method for classifying text blocks in a data collection response, comprising: (a) Receiving a data collection response fetched from a data collection target in accordance with a data collection request via a proxy server, wherein the fetching agent (106) receives the request from the user equipment that issued the request, and the fetching agent executes the request for the data collection target via the proxy server; (b) Extract multiple text blocks from the data collection response; For each of the plurality of text blocks: (c) Obtain the path to the location of the corresponding text block within the data collection response; (d) Collect meta-attributes within the data collection response via a path that describe the display and functional characteristics of a text block within the page specified by the data collection response, wherein the meta-attributes include style attributes selected from at least one of the following: text block position, text block color, text block font style, text block font size, and whether the text block is underlined; (e) Derive category attributes from text blocks, paths and meta attributes, wherein, before exporting the category attributes, irrelevant text blocks are removed from the plurality of text blocks, the removal including removing text blocks with style attributes that indicate that the text blocks are not visible within the page; (f) Perform a trained machine learning classification model on the classification attribute to determine the classification of the text block, wherein the determined classification of the text block is selected from a plurality of predefined content categories, the plurality of predefined content categories corresponding to different data fields of a structured list of items and including at least a price category, a title category and a description category; (g) Construct a dataset using the plurality of text blocks and the corresponding classification determined in (f) for each of the plurality of text blocks; and (h) Transmit the dataset to the user equipment that issued the request. The method further includes preprocessing data points to derive categorical attributes from associated meta attributes, and assigning associated categorical attributes to corresponding text blocks, wherein the data points include text blocks within the data collection response, associated meta attributes, corresponding paths, and source HTML elements, and the categorical attributes assigned to the text blocks are derived from at least one of the following: HTML tags, classes, groups of ids and variables of the text block, text attributes of the text block, style attributes of the text block, and paths.

2. The method of claim 1, wherein the received data collection response is in HTML format.

3. The method of claim 1, wherein the received data collection response is in MHTML format.

4. The method of claim 3, further comprising rendering a data collection response in MHTML format to extract an HTML file, wherein extraction (b) includes extracting the plurality of text blocks from the HTML file.

5. The method of claim 4, wherein collection (d) includes collecting meta attributes from the HTML and non-HTML portions of the data collection response.

6. The method of any one of claims 1-5, wherein the identified classification attributes are registered and assigned to text blocks in a numerical format.

7. The method of any one of claims 1-5, further comprising concatenating individual text blocks from the plurality of text blocks.

8. The method of any one of claims 1-5, wherein performing (f) includes applying a classification attribute to a plurality of machine learning classification models, each of the plurality of machine learning classification models being trained to identify whether a block of text belongs to a category.

9. The method of claim 8, wherein each of the plurality of machine learning classification models determines a classification probability, the classification probability indicating the likelihood that a text block belongs to a category that the corresponding machine learning classification model was trained to detect.

10. The method of any one of claims 1-5, wherein the trained machine learning classification model employed is at least one of the following, but not limited to: bag-of-words, Naive Bayes algorithm, support vector machine, logistic regression, random forest classifier or extreme gradient boosting model.

11. The method of any one of claims 1-5, wherein the classification decision is submitted to the classification platform for quality assurance, wherein the assigned classification is checked and verified through crowdsourcing.

12. The method of claim 11, wherein the quality-assured classification decision is classified as correct and becomes part of the training of the future machine learning classification model and is incorporated into the corresponding training set.

13. The method of any one of claims 1-5, wherein the data collection response includes non-textual information.

14. The method of claim 13, wherein the content delivered within non-textual information is processed by a trained machine learning classification model.

15. The method of any one of claims 1-5, wherein the transmission (h) is performed via an intermediary component.

16. The method of claim 15, wherein, The intermediary component is a crawling proxy.

17. The method of any one of claims 1-5, wherein if extraction (b) does not return a recognizable text block, then a data collection response is not submitted for classification, and the request is resubmitted as a data collection request.

Images