Webpage data crawling method and device, electronic equipment and medium

Abstract

The invention discloses a webpage data crawling method and device, electronic equipment and a medium, and the method comprises the steps: analyzing a data crawling request input by a user, and obtaining an analysis result; performing data crawling based on the data crawling request to obtain to-be-cleaned webpage data; according to an analysis result, determining whether a target cleaning strategy capable of being used for cleaning the to-be-cleaned webpage data exists in a data cleaning strategy library or not; if the to-be-cleaned webpage data exists, cleaning the to-be-cleaned webpage data through a target cleaning strategy to obtain target crawling data; if not, generating a target cleaning strategy based on the target large model according to the to-be-cleaned webpage data, and executing the step of cleaning the to-be-cleaned webpage data through the target cleaning strategy to obtain target crawling data. Therefore, through intervention of the target large model, the cleaning strategy is automatically generated, the strategy generation efficiency and accuracy are improved, the high cost caused by dependence on manual maintenance of the cleaning strategy is avoided, and then the cleaning efficiency and precision of the webpage data are improved.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a method, apparatus, electronic device and medium for crawling web page data. Background Technology

[0002] With the rapid development of internet technology, the amount of information on the web is growing exponentially, covering data from various fields such as news, social media, and e-commerce. Therefore, effectively cleaning the crawled web data to accurately extract the target data needed by users is crucial.

[0003] Currently, data cleaning of crawled web pages mainly relies on pre-defined cleaning strategies. However, in actual web crawling, some web page requests may not have pre-defined cleaning strategies, resulting in low data crawling accuracy and impacting user experience. Furthermore, due to significant differences in website structure across various data sources, different websites, and even the same website, require manually customized cleaning strategies based on the website structure, leading to low data cleaning efficiency and high costs.

[0004] Therefore, how to improve the accuracy and efficiency of web page data cleaning and reduce the cost of data cleaning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, one aspect of this application provides a method for crawling web page data, the method comprising:

[0006] Parse the user's input data crawling request to obtain the parsing results;

[0007] Data is crawled based on the data crawling request to obtain the web page data to be cleaned;

[0008] Based on the analysis results, determine whether there is a target cleaning strategy in the pre-built data cleaning strategy library that can be used to clean the web page data to be cleaned;

[0009] If it exists, the webpage data to be cleaned is cleaned using the target cleaning strategy to obtain the target crawled data;

[0010] If it does not exist, the target cleaning strategy is generated based on the target big model according to the web page data to be cleaned, and the step of cleaning the web page data to be cleaned by the target cleaning strategy to obtain the target crawled data is executed.

[0011] Optionally, parsing the user-input data crawling request to obtain the parsing result includes:

[0012] Determine the Uniform Resource Locator and request text corresponding to the data crawling request;

[0013] Perform semantic parsing on the request text to obtain keywords from the text;

[0014] Based on the keywords, determine the data cleaning dimension corresponding to the data crawling request; wherein, the data cleaning dimension is the data type of the keywords;

[0015] The data request dimension and the Uniform Resource Locator are vectorized to obtain a target vector; and the target vector is used as the parsing result.

[0016] Optionally, the step of cleaning the webpage data to be cleaned using the target cleaning strategy to obtain the target crawled data includes:

[0017] The web page data to be cleaned is parsed to obtain web page structure data;

[0018] Extract the DOM structure that conforms to the target cleaning strategy from the webpage structure data;

[0019] Locate the DOM node in the DOM structure that corresponds to the data cleaning dimension;

[0020] Extract the data content corresponding to the DOM node as the target crawling data.

[0021] Optionally, based on the webpage data to be cleaned, a target cleaning strategy is generated based on the target large model, including:

[0022] The web page data to be cleaned is parsed to obtain web page structure data;

[0023] Determine the hierarchical structure of webpage structure tags within the corresponding webpage structure data to obtain the positioning tags;

[0024] Based on the positioning tags, extract the data information wrapped by each of the webpage structure tags to obtain webpage tag data;

[0025] The credibility score of each webpage tag data is determined using the target large model; the credibility score is used to characterize the credibility of the webpage tag data in obtaining the target crawled data, and the higher the credibility score, the higher the credibility.

[0026] The target cleaning strategy is generated based on the target webpage tag data; the target webpage tag data is data whose credibility score is greater than a threshold.

[0027] Optionally, determining the credibility score of each webpage tag data using the target large model includes:

[0028] Obtain the data content and tag attributes of the webpage tag data, and determine the data type of the data content;

[0029] Using the target large model, the semantic score and attribute score of the webpage tag data are determined; the semantic score is used to characterize the degree of semantic relevance between the data content and the data type; the attribute score is used to characterize the degree of attribute relevance between the tag attributes and the data type.

[0030] Assign corresponding weight coefficients to the semantic score and the attribute score;

[0031] The credibility score is obtained by weighting and summing the semantic score and the attribute score according to the weight coefficients.

[0032] Optionally, after obtaining the credibility score by weighted summation of the semantic score and the attribute score according to the weight coefficients, the process includes:

[0033] Obtain feedback signals regarding the credibility score;

[0034] When the feedback signal indicates that the credibility score is inaccurate, the weight coefficient is adjusted and / or the number of attribute scores is increased; and the process returns to the step of determining the semantic score and attribute score of the webpage tag data through the target large model; wherein, different attribute scores correspond to different tag attributes; and the weight coefficient of the semantic score is greater than the weight coefficient of the attribute score.

[0035] Optionally, generating the target cleaning strategy based on the target webpage tag data includes:

[0036] The target positioning tag for obtaining the target webpage tag data;

[0037] Determine the largest common container label for a specified location label group; the specified location label group consists of target location labels corresponding to different data types.

[0038] Remove the specified common container tag from the largest common container tag; the specified common container tag exists only once in each of the web page data to be cleaned.

[0039] The container label located at the innermost layer of the wrapping hierarchy among the largest common container labels is taken as the target container label;

[0040] The target cleaning strategy is generated based on the webpage tag data corresponding to the target container tag.

[0041] Optionally, after cleaning the webpage data to be cleaned using the target cleaning strategy to obtain the target crawling data, the process includes:

[0042] Determine whether the target crawled data includes multimodal data other than text data;

[0043] If so, semantic parsing is performed on the multimodal data using a specified large model to obtain parsed text; the text data, the parsed text, and the multimodal data are then structured to obtain the target output data.

[0044] Another aspect of this application provides a web page data crawling apparatus, the apparatus comprising:

[0045] The request parsing module is used to parse the data crawling request input by the user and obtain the parsing result;

[0046] The data crawling module is used to crawl data based on the data crawling request to obtain the web page data to be cleaned;

[0047] The strategy determination module is used to determine, based on the parsing results, whether there exists a target cleaning strategy in the pre-built data cleaning strategy library that can be used to clean the web page data to be cleaned; if it exists, the data cleaning module is called; if it does not exist, the strategy generation module is called first, and then the data cleaning module is called.

[0048] The data cleaning module is used to clean the web page data to be cleaned using the target cleaning strategy to obtain the target crawled data.

[0049] The strategy generation module is used to generate the target cleaning strategy based on the target big model according to the web page data to be cleaned.

[0050] Another aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the web page data crawling method.

[0051] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the steps of the web page data crawling method being executed by a processor.

[0052] The web page data crawling method, apparatus, electronic device, and medium provided in this application have the following beneficial effects: based on the crawled web page data to be cleaned, a target cleaning strategy that can be used to clean the web page data to be cleaned is automatically generated through the intervention of a target large model, thereby improving the efficiency and accuracy of cleaning strategy generation, avoiding the high cost caused by relying on manual maintenance of cleaning strategies, and thus improving the efficiency and accuracy of web page data cleaning, and ensuring the user's data crawling experience. Attached Figure Description

[0053] Figure 1 A flowchart illustrating a web page data crawling method provided in an embodiment of this application;

[0054] Figure 2 A schematic diagram illustrating the principle of a web page data crawling method provided in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of a data crawling process provided in an embodiment of this application;

[0056] Figure 4 A flowchart illustrating a web page data crawling method provided in another embodiment of this application;

[0057] Figure 5 A schematic diagram illustrating the webpage effect of a container tag provided in an embodiment of this application;

[0058] Figure 6 This is a schematic diagram illustrating the webpage effect of another container tag provided in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the structure of a web page data crawling device provided in an embodiment of this application;

[0060] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0061] The reference numerals in the attached diagram are as follows: 70 is the request parsing module, 71 is the data crawling module, 72 is the strategy determination module, 73 is the data cleaning module, 74 is the strategy generation module, 80 is the memory, 81 is the processor, 82 is the display screen, 83 is the input / output interface, 84 is the communication interface, 85 is the power supply, 86 is the communication bus, 801 is the computer program, 802 is the operating system, and 803 is the data. Detailed Implementation

[0062] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0063] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0064] Figure 1 This is a flowchart illustrating a web page data crawling method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0065] S10: Parse the user's data crawling request and obtain the parsing result;

[0066] Figure 2 This is a schematic diagram illustrating the principle of a web page data crawling method provided in an embodiment of this application, as shown below. Figure 2 As shown, in a specific embodiment, when a user has a data crawling need, the user inputs a data crawling request on the user's end, and the data crawling request is sent from the user's end to the request processing end. It should be noted that, in an optional embodiment, the data crawling request may include information such as the Uniform Resource Locator (URL) of the target website, the data type to be crawled (e.g., text, images, and tables), the characteristics of the page elements where the data is located (e.g., HyperText Markup Language (HTML) tags and class names), and the crawling scope (e.g., a single page or multiple pages).

[0067] After receiving the user's data crawling request, the request processing end parses the request to obtain the parsing results. These results include, but are not limited to, the URL and data cleaning dimensions. Data cleaning dimensions can be understood as the data types that need to be crawled. Therefore, the parsing results can be used as the basis for subsequent webpage data cleaning.

[0068] It is worth noting that, in an alternative embodiment, such as Figure 2As shown, in order to ensure the accuracy and efficiency of data crawling, the request processing end can parse the data crawling request through the target large model.

[0069] It should be noted that, in specific embodiments, when a user inputs a data crawling request through the user terminal, they can input text data, or input it in the form of a form. Of course, they can also input it in the form of voice, etc. This application does not limit this.

[0070] In one optional embodiment, when collecting user data through a form, the data processing requirements entered into the form need to be converted into JSON format. Similarly, the form includes, but is not limited to, URLs and data processing dimensions.

[0071] S11: Based on the data crawling request, crawl the data to be cleaned.

[0072] While parsing the data crawling requests, data is crawled based on the requests to obtain the webpage data to be cleaned. In the data cleaning process, the crawler program is launched, and the crawler accesses the target website according to the URL in the request, thereby obtaining the webpage data to be cleaned.

[0073] Figure 3 This is a schematic diagram of a data crawling process provided in an embodiment of this application. In an optional embodiment, such as... Figure 3 As shown, website data scraping can be performed using the Puppeteer library, which is based on a Node environment. Specifically, the data scraping provided in this application runs in a Node environment and is based on a deeply customized version of Puppeteer. Puppeteer provides high-level APIs to control Chromium via the DevTools protocol. Therefore, most operations that can be performed manually in a browser can be accomplished using Puppeteer.

[0074] In one alternative embodiment, such as Figure 3As shown, modules developed based on Puppeteer can include Page WaitUntil, Page Screenshot, AutoScroll, FilterTags, CaptureImage, and CaptureVideo. The Page WaitUntil module primarily addresses the issue of incomplete or failed page information retrieval caused by dynamically loaded web pages (i.e., single-page applications). Due to the current front-end / back-end separation architecture and the maturity of front-end frameworks, page rendering is typically done dynamically using JavaScript, and the page URL does not change frequently. This is significantly different from traditional server-side rendering of complete pages (i.e., template engines returning pages to the browser). Therefore, in the Page WaitUntil module, meeting the following three conditions indicates that the web page has finished loading and meets the requirements for web page retrieval.

[0075] The specific conditions include: Condition 1 is that the page's Load event has been triggered, meaning it will only trigger after all resources have finished loading. Condition 2 is that the page's DOMContentLoaded event has been triggered, meaning it will trigger after the HTML document has finished loading and the DOM tree has been built. Condition 3 is that it will trigger when there is no longer a network connection, and after a preset time (e.g., 500 milliseconds).

[0076] The Page Screenshot module is used to archive screenshots of crawled web pages, facilitating later tracking of crawling events and evaluation of the training quality of large models. The AutoScroll module encapsulates a series of JavaScript execution functions that simulate user scrolling of web pages. In one optional embodiment, the AutoScroll module can provide three optional parameters, specifically including the class name of the "Load More" button, the number of scroll loads, and the callback function triggered after loading is complete.

[0077] In specific implementations, loading more content on a webpage typically involves two triggering methods: scrolling and clicking a button. If the webpage loads via scrolling, there's no need to pass in the class name of the "Load More" button; Puppeteer will execute the encapsulated AutoScroll function via page.evaluate to simulate the user's scrolling action and load more content. If the webpage loads via clicking a "Load More" button, page.click will simulate the user's click action and load more content. Regardless of the loading method, a callback function indicating that the webpage has finished loading will be triggered at the end, facilitating the smooth flow of subsequent steps.

[0078] The `FilterTags` module is used to clean up invalid tags, a process known as "pre-cleaning." It cleans up invalid Document Object Model (DOM) nodes by manually passing in tag class names and keywords and using regular expressions. The `CaptureImage` module is used to capture image files from web pages, and the `CaptureVideo` module is used to capture video files from web pages.

[0079] like Figure 3 As shown, after processing by the above six modules, the webpage ultimately yields HTML files, image files, and video files. In an optional embodiment, the HTML file needs to be converted into a Markdown file using the Turndown module in NodeJS to prepare for subsequent processing of the target large model.

[0080] S12: Based on the parsing results, determine whether there is a target cleaning strategy in the pre-built data cleaning strategy library that can be used to clean the web page data to be cleaned; if it exists, proceed to step S13; if it does not exist, proceed to step S14.

[0081] S13: Clean the webpage data to be cleaned using the target cleaning strategy to obtain the target crawling data;

[0082] S14: Based on the webpage data to be cleaned, generate the target cleaning strategy based on the target large model, and then execute step S13.

[0083] Furthermore, after parsing the data crawling request, such as... Figure 2 As shown, the parsing results are matched with the strategies in the pre-built data cleaning strategy library. If there is a target cleaning strategy in the cleaning strategy library that can be used to clean the currently acquired web page data to be cleaned, the target cleaning strategy is directly retrieved from the strategy library to clean the web page data to be cleaned, thereby obtaining the target crawled data.

[0084] However, in another optional embodiment, if the target cleaning strategy does not exist in the current data cleaning strategy library, a target cleaning strategy is automatically generated based on the crawled webpage data to be cleaned and the target large model. After generation, the webpage data to be cleaned is requested using the target cleaning strategy, and the automatically generated target cleaning strategy is stored in the data cleaning strategy library. Specifically, in one optional embodiment, the target large model can analyze the webpage data to be cleaned, identify its data structure and characteristics, and thus automatically generate a cleaning strategy suitable for the current data.

[0085] It should be noted that the strategies in the pre-built data cleaning strategy library can be automatically generated by the target large model, or they can be customized manually based on past data cleaning experience. For example, for crawling text content from news websites, there might be cleaning strategies such as removing advertising information, extracting the main text of the article, and formatting dates. This application does not limit the storage method of the cleaning strategies in the data cleaning strategy library, but it should be noted that the data cleaning strategy library includes the correspondence between cleaning strategies and URLs to facilitate quick location of cleaning strategies based on URLs later.

[0086] In addition, it should be noted that, in an optional embodiment, the cleaning process may include operations such as removing HTML tags, extracting specific text or data fields, filtering out irrelevant noise data, and formatting and normalizing the data.

[0087] In specific embodiments, the target large model may include, but is not limited to, the GPT (Generative Pre-trained Transformer) series, the BERT model, large models based on the Transformer architecture, LLaMA (Large Language Model Meta AI), and Tongyi Qianwen. This application does not limit the target large model.

[0088] Therefore, the web page data crawling method provided in this application automatically generates a target cleaning strategy that can be used to clean the web page data to be cleaned by the intervention of a target large model, based on the crawled web page data to be cleaned. This improves the efficiency and accuracy of cleaning strategy generation, avoids the high cost caused by relying on manual maintenance of cleaning strategies, and thus improves the efficiency and accuracy of web page data cleaning, ensuring the user's data crawling experience.

[0089] In one optional embodiment, parsing the user-inputted data crawling request to obtain the parsing result includes:

[0090] Determine the Uniform Resource Locator and request text corresponding to the data crawling request;

[0091] Perform semantic parsing on the request text to extract keywords from it;

[0092] Based on the keywords, determine the data cleaning dimensions corresponding to the data crawling request; where the data cleaning dimension is the data type of the keyword.

[0093] The data request dimension and Uniform Resource Locator are vectorized to obtain the target vector; and the target vector is used as the parsing result.

[0094] In a specific embodiment, when parsing a user-inputted data crawling request, it is first necessary to clarify the data source of the crawling request, that is, to determine the URL corresponding to the data crawling request. A URL is an address used on the Internet to identify a resource. It includes the protocol type, the IP address (or domain name) of the host storing the resource, and the address of the resource itself, such as a file path. For example, a URL might be www.aaa.com.

[0095] After determining the URL, semantic parsing is performed on the request text. The request text can be a natural language description input by the user or a structured query statement; this application does not limit this. In an optional embodiment, the text is parsed using a target big model to obtain keywords. It is understood that keywords are core to understanding user needs and are also an important condition for determining the target cleaning strategy.

[0096] To make it easier to understand, an example will be given below. For instance, if the user enters the request text "Get the air quality of major cities around the world in 2024", after parsing the request text, the keywords may include "2024", "major cities around the world", "air quality data", etc.

[0097] In a specific embodiment, after determining the keywords, the next step is to determine the data cleaning dimension corresponding to the data crawling request based on the keywords, that is, to determine the data type to which the keywords belong. For example, when a user visits a URL for an e-commerce website and the keyword is "power bank," the corresponding data cleaning dimension is the product name.

[0098] To facilitate matching the desired target cleaning strategy within the data cleaning strategy library, the determined URLs and data cleaning dimensions are vectorized. Specifically, vectorization is the process of converting text or data into numerical vectors for easier computer processing and analysis. For example, in one alternative embodiment, word embedding techniques (e.g., Word2Vec or GloVe) can be used to convert the data request dimension into a vector, while the various components of the URL (e.g., protocol, domain name, path) can also be encoded into vectors separately.

[0099] The vector corresponding to the data source request dimension and the vector corresponding to the URL are combined to obtain the target vector. The target vector is then used as the parsing result so that the target cleaning strategy can be matched in the data cleaning strategy library based on the vector.

[0100] In one optional embodiment, the webpage data to be cleaned is cleaned using a targeted cleaning strategy to obtain the target crawling data, including:

[0101] The webpage structure data to be cleaned is parsed to obtain the webpage structure data;

[0102] Extract DOM structures that conform to the target cleaning strategy from web page structure data;

[0103] Locate the DOM node in the DOM structure that corresponds to the data cleaning dimension;

[0104] Extract the data content corresponding to the DOM nodes and use it as the target data to crawl.

[0105] In the data cleaning process, the first step is to parse the webpage structure of the data to be cleaned. The purpose of parsing the webpage structure is to convert the HTML code of the webpage into a manipulable DOM tree structure. The DOM tree is a hierarchical node structure, where each node represents an element in the webpage (e.g., ...). , , (etc.). By analyzing the structure of web pages, we can clarify how the pages are organized, providing a foundation for subsequent data extraction and cleaning.

[0106] Furthermore, using webpage structure data as a data source, the DOM structure that conforms to the target cleaning strategy is extracted. For ease of understanding, an example is given using product names as the data cleaning dimension in the cleaning strategy.

[0107] For example, the target cleaning strategy is:

[0108] "label":"Product Name",

[0109] "key":"productName",

[0110] "matches":[

[0111] {

[0112] "value":"product-name",

[0113] "valueType":"text",

[0114] "type":"className",

[0115] "source":1

[0116] }

[0117] Here, `label` is the display name of the cleaned data, i.e., the data cleansing dimension; in this example, it's the product name. `key` is the core identifier, i.e., the core representation of the data cleansing dimension, and also the core identifier in the program. The display name `label` can be variable; for example, the product name label can also be "product name," "product name," etc., but the corresponding meaning of `key` remains constant. Subsequent retrieval strategies for the product name field will use `key` as the primary retrieval basis.

[0118] `matches` represents specific matching rules, including `matches[i].value`, `matches[i].valueType`, `matches[i].type`, and `matches[i].source`. `matches[i].value` is the match value for a specific matching rule, and `matches[i].valueType` is the type of the match value. Different types result in different matching logic. For example, `text` represents exact text matching; when `matches[i].valueType` is of type `text`, the matching logic is that a match is successful only if the value to be matched is exactly equal to `matches[i].value`. `regular` represents regular expression matching; when `matches[i].valueType` is of type `regular`, `matches[i].value` is a regular expression, and a match is successful only if the value to be matched satisfies the regular expression in `matches[i].value`. `include` represents the type of matching that includes text; when `matches[i].valueType` is of type `include`, the matching logic is that a match is successful only if the value to be matched contains the value in `matches[i].value`.

[0119] `matches[i].valueType` represents the webpage structure information to be matched. The type of `Type` can be continuously expanded to support more modal data input and matching. For example, `class` uses the `class` attribute of the webpage's DOM structure tag as the match value. `interValue` uses the value wrapped in the webpage's DOM tag structure as the match value. `ID` uses the `ID` attribute of the webpage's DOM structure tag as the match value. `imgUrl` uses an image link as the match value.

[0120] `matches[i].source` represents the credibility score, which indicates the degree of credibility of the target crawled data when the current data cleaning strategy is applied. The higher the credibility score, the higher the credibility.

[0121] Based on the above examples, in a specific embodiment, when cleaning the web page data to be cleaned, the DOM structure that satisfies the target cleaning strategy is first extracted from a large amount of web page structure data. Specifically, the DOM structure in which the attribute value of the class attribute (determined by "type":"className" in the above example) is equal to product-name (determined by "valueType":"text" in the above example) is extracted.

[0122] Furthermore, locate the DOM nodes in the DOM structure that correspond to the data cleaning dimension, and extract the data content corresponding to the DOM nodes as the target data to be crawled, that is, take the product name data as the target data to be crawled.

[0123] Figure 4 This is a flowchart illustrating a web page data crawling method provided in another embodiment of this application, as shown below. Figure 4 The process of generating a target cleaning strategy based on a target large model, according to the webpage data to be cleaned, includes:

[0124] S40: Perform webpage structure parsing on the webpage data to be cleaned to obtain webpage structure data;

[0125] S41: Determine the hierarchical structure of the webpage structure tags in the corresponding webpage structure data to obtain the positioning tags;

[0126] It is understandable that data cleaning strategies are closely related to webpage structure. Therefore, in specific embodiments, when automatically generating the target cleaning strategy, the webpage structure data to be cleaned crawled by the crawler is first parsed to obtain webpage structure data.

[0127] For example, the webpage structure data is as follows:

[0128]

[0129]

[0130] <title> Product Details< / title>

[0131]

[0132]

[0133] <div class＝"product-info">

[0134] <p class＝"name"> 20000mAh high-power power bank

[0135] <div class＝"price-box">

[0136] <p class＝"price">￥128

[0137]

[0138]

[0139]

[0140]

[0141] Furthermore, after parsing the webpage structure data, it is necessary to determine the hierarchical structure of the webpage structure tags within the corresponding webpage structure data, thereby obtaining the positioning tags. Webpage structure tags are the markup symbols in an HTML document that construct the webpage structure data; for example, the `<head>`, `<body>`, `<body>`, and `<body>` tags mentioned above. , These are all webpage structure tags.

[0142] Hierarchical structure refers to the positional relationship of each webpage structural tag within the webpage structural data. In a specific embodiment, by determining the hierarchical structure of webpage structural tags within the webpage structural data, the hierarchical relationship and location between each webpage structural tag can be quickly identified during the subsequent cleaning strategy generation process. In an optional embodiment, positioning tags are used to characterize the hierarchical relationship of webpage structural tags within the corresponding webpage structural data. Furthermore, in an optional embodiment, the positioning tags can be an array or other forms; this application does not limit the specific form of the positioning tags.

[0143] S42: Based on the positioning tags, extract the data information wrapped by each webpage structure tag to obtain the webpage tag data;

[0144] Furthermore, based on the location tags, data information wrapped in webpage structure tags can be extracted to obtain a webpage tag dataset. For ease of understanding, the data information wrapped in webpage structure tags will be explained using the examples above. For instance, regarding " <p class＝"name"> 20000mAh high-power power bank "Web page structure tags are" The data information enclosed by the two p tags includes class="name" and 20000mAh high-power power bank.

[0145] To facilitate understanding, the following example illustrates the webpage tag data generated from the webpage structure data described above.

[0146] For example, the webpage tag data generated based on the above webpage structure data is as follows:

[0147]

[0148]

[0149] In this array, `tag` represents the webpage structure tag, and `index` represents the positioning tag. The position of each item in the array represents the structural level of the webpage structure tag, and the specific number represents the specific order of the tag in that level. For example, the first item, `tag`, is the data item for `html`, that is, the webpage structure tag is `html`. Since the webpage structure tag is located at the outermost level in the webpage structure data, the positioning tag `index` is an array of 1 elements [x]. Since `html` is the first item in this level, the positioning tag `index` is [1].

[0150] Similarly, the 20000mAh high-power power bank belongs to... The tags, in hierarchical structure, are: tag, tag, ... <div class＝"product-info">Tags and <p class＝"name"> Therefore, with a level of 4, the array structure for the locating tag index should be [x, x, x, x]. Since the tag's level index is 1, and the index of the tag's location is 2, <div class＝"product-info">The hierarchical index is 1. <p class＝"name">The hierarchical index is 1, therefore the location label index is [1, 2, 1, 1].

[0151] The `attributes` attribute contains the tag's attribute information, including the `attrName` tag attribute and its corresponding `value` tag attribute value. The last `value` is the tag's actual display value. It's worth noting that when a tag in the webpage structure does not have a specific display value, the last `value` will be empty. For example, in the example above, the tag has no specific display value, therefore, `value` is an empty string.

[0152] S43: Determine the credibility score of each webpage tag data through the target large model; the credibility score is used to characterize the credibility of webpage tag data for obtaining target crawling data. The higher the credibility score, the higher the credibility.

[0153] S44: Generate a target cleaning strategy based on the target webpage tag data; the target webpage tag data consists of data with a credibility score greater than the threshold.

[0154] Furthermore, the credibility score is assigned to each webpage tag data using the target large model. A higher credibility score indicates a higher degree of credibility for the corresponding webpage tag data in obtaining target crawling data. To improve data crawling accuracy, data with credibility scores greater than a threshold (e.g., a threshold of 0.7) are extracted as target webpage tag data, and a target cleaning strategy is generated based on the target webpage tag data.

[0155] Based on the above embodiments, as an optional embodiment, the credibility score of each webpage tag data is determined through a target large model, including:

[0156] Obtain the data content and tag attributes of webpage tag data, and determine the data type of the data content;

[0157] Using the target large model, we determine the semantic score and attribute score of webpage tag data; the semantic score is used to characterize the semantic relevance between data content and data type; the attribute score is used to characterize the attribute relevance between tag attributes and data type.

[0158] Assign corresponding weight coefficients to semantic scores and attribute scores;

[0159] The credibility score is obtained by weighting and summing the semantic score and attribute score based on the weight coefficients.

[0160] In a specific embodiment, the data content, tag attributes, and data type corresponding to the webpage tag data are obtained. For example, in the example of webpage tag data in the above embodiment, one of the webpage tag data is:

[0161]

[0162] In this webpage tag data, the data content is "20000mAh high-power power bank," and the tag attribute `attrName` is "class," corresponding to the data type of "product name." Furthermore, using a target large model, the semantic relevance between the data content and data type in the webpage tag data is scored, resulting in a semantic score. In a specific embodiment, a higher semantic score indicates a higher degree of relevance between the data content and data type.

[0163] For example, given data such as "20000mAh high-power power bank" and "product name" as the data type, semantic scoring can be performed using a target large model. Specifically, the generated input text is "Evaluate the probability that the following text is a product name, with a minimum of 0 representing completely impossible and a maximum of 1 representing completely possible, where the evaluation text is '20000mAh high-power power bank'". Then, the target large model performs inference to provide the final semantic score result.

[0164] Furthermore, the relevance between tag attributes and data types in webpage tag data is scored using a target large-scale model, resulting in attribute scores. Specifically, the evaluation process calculates scores based on tag attribute values ​​and data types; that is, attribute scores are based on attribute values, and the score result represents the relevance between tag attributes and data types. In other words, when scoring using the target large-scale model, the input is the tag attribute value and data type, and the final result reflects the relevance between the tag attribute `attrName` and the data type. An example will be provided below for easier understanding.

[0165] For example, in the above example, the tag attribute `attrName` is `class`, and the corresponding attribute value of the `class` tag is `name`, with the data type being "product name". In a specific embodiment, the relevance between "name" and "product name" is evaluated using a target large model.

[0166] Furthermore, to improve the accuracy of the credibility score, in one optional embodiment, corresponding weight coefficients are assigned to the semantic score and the attribute score, and a weighted sum is performed based on the assigned weight coefficients to obtain the final credibility score. For example, in one optional embodiment, the weight coefficient corresponding to the semantic score is 1, and the weight coefficient corresponding to the attribute score is 0.2, then the credibility score = 1 * semantic score + 0.2 * attribute score.

[0167] Based on the above embodiments, as an optional embodiment, after obtaining the credibility score by weighted summation of semantic scores and attribute scores according to weight coefficients, the following steps are included:

[0168] Obtain feedback signals regarding the credibility score;

[0169] When the feedback signal does not accurately represent the credibility score, adjust the weight coefficients and / or increase the number of attribute scores; and return to the steps of determining the semantic score and attribute score of the webpage tag data through the target large model; wherein, different attribute scores correspond to different tag attributes; the weight coefficient of the semantic score is greater than the weight coefficient of the attribute score.

[0170] To ensure the accuracy of the target cleaning strategy, after calculating the credibility score, it can be re-verified using a large target model or manually. At this point, feedback signals regarding the credibility score are obtained. If the feedback signal is positive, the current credibility score result is used as the final score for the webpage tag data.

[0171] If the feedback signal does not accurately represent the credibility score, i.e., the feedback signal is not positive, the weighting system can be adjusted, and / or the number of attribute scores can be increased. When adjusting the coefficients, it is necessary to ensure that the weighting coefficient of the semantic score is greater than the weighting coefficient of the attribute score.

[0172] It is understandable that, in specific embodiments, there are many types of tag attributes. By increasing the amount of tag data, the weighted summation term for calculating the credibility score can be increased, thereby improving the accuracy of the final credibility score.

[0173] Therefore, in order to ensure the accuracy of the target cleaning strategy, the web page data crawling method provided in this application requires providing users with the logic of cleaning strategy and corresponding cleaning data. Users need to manually confirm and filter the cleaning strategy.

[0174] Based on the above embodiments, as an optional embodiment, a target cleaning strategy is generated according to the target webpage tag data, including:

[0175] Targeted tags for obtaining target webpage tag data;

[0176] Determine the largest common container label for the specified location label group; the specified location label group consists of target location labels corresponding to different data types;

[0177] Remove specified common container tags from the largest common container tag; the specified common container tag must exist only once in each webpage data to be cleaned.

[0178] Take the innermost container label in the wrapper hierarchy from the largest common container labels as the target container label;

[0179] A target cleaning strategy is generated based on the webpage tag data corresponding to the target container tag.

[0180] It is understandable that each data request can obtain a large amount of web page tag data. Based on the credibility score being greater than the threshold, multiple target web page tag data can be filtered out for different data types. In a specific embodiment, the target positioning tags of the target web page tag data are extracted.

[0181] Furthermore, to determine the webpage tag data that can be used for crawling the target data, the largest common container tag of the specified positioning tag group is determined. This specified positioning tag group consists of target positioning tags corresponding to different data types. The largest common container tag refers to the nearest common parent element that can be found when tracing back up the DOM tree for multiple target positioning tags corresponding to different data types. This parent element is the largest common container of the specified positioning tag group. It can also be understood as the largest common container tag being the nearest common ancestor element of the specified positioning tag group in the DOM tree. For ease of understanding, an example will be provided below.

[0182] For example, the data types include three categories: product name, product price, and product reviews. The specified positioning tag groups include: the target positioning tag index for the product name is [1, 2, 1, 1], the target positioning tag index for the product price is [1, 2, 1, 2, 1], and the target positioning tag index for the product reviews is [1, 2, 1, 3, 3]. In this example, the largest common container tag is [1, 2, 1], and based on this tag, the data for this webpage tag can be retrieved in reverse.

[0183] It should be noted that container tags are tags used to wrap a set of data content. In the data cleaning process, they are used to determine the scope of data acquisition, that is, to define the boundaries of the data. Figure 5 This is a schematic diagram of a webpage effect of a container tag provided in an embodiment of this application. For ease of understanding, it will be described below in conjunction with... Figure 5 Please provide an explanation.

[0184] For example, in Figure 5 In the context of a dataset consisting of product image 1, product price 1, and product review 1, after determining its largest common container label, as follows: Figure 5 The dashed box in the image can be used to define data boundaries during the data crawling process, thereby allowing a complete set of data to be displayed to the user on the page.

[0185] In specific embodiments, it is understood that any different specified location tag groups can determine a maximum common container tag. Therefore, in specific embodiments, it is necessary to filter the maximum common container tag. Figure 6 This is a schematic diagram illustrating the webpage effect of another container tag provided in an embodiment of this application, such as... Figure 6 As shown, when determining the largest common container label for product image 1, product price 2, and product review 3, what is obtained is the outermost container of all products (i.e., Figure 6 (As shown by the dashed box), it is clear that in a specific embodiment, during data cleaning, Figure 6 The results shown need to be discarded.

[0186] In one optional embodiment, after determining the largest common container tag for each specified positioning tag group, the largest common container tag that exists only once in the web page data to be cleaned is removed. For example, if all the content displayed on a page includes... Figure 6 The products shown are 1-3, and the largest common container label is the outermost dashed box. Obviously, with this data as the boundary, there is only one largest common container label in each page. At this time, the largest common container label needs to be removed.

[0187] In specific web page data crawling, each web page contains multiple data content groups. Therefore, the largest common container tag corresponding to the data content groups is the same, for example, such as Figure 5 As shown, Product 1, Product 2, and Product 3 can all be wrapped and cleaned using the largest common container tag indicated by the dashed box. In other words, products with the same largest common container tag appear repeatedly in the same webpage data to be cleaned.

[0188] Furthermore, it is understood that there is a wrapping relationship between different maximum common container labels. In an optional embodiment, the container label located at the innermost layer of the wrapping level among the maximum common container labels is taken as the target container label, that is, the container labels outside the innermost layer are removed.

[0189] For example, if Figure 6 The largest common container label A shown is [1, 2, 1], while Figure 5 The largest common container label B shown is [1, 2, 1, 3]. According to the design principles of indexes, container A encloses container B, meaning the data boundary of container A is larger than the data boundary of container B. Therefore, container A needs to be removed, and container B retained. Figure 6 The largest common container label shown is removed, and the remaining ones are retained. Figure 5 The largest common container label shown.

[0190] In a specific embodiment, after filtering and screening using the largest common container tag, the final target container tags do not have a hierarchical relationship with each other, that is, there is no inclusion relationship, and the target container tags are containers that appear cyclically in the web page data to be cleaned. Therefore, the target container tags containing data such as product names and prices represent data from different dimensions. For example, some pages contain both product list information and system-recommended product list information, or some stock pages contain multiple stock market charts. During data extraction, users can choose the data they are interested in to extract.

[0191] Furthermore, after obtaining the target container tags, a target cleaning strategy can be generated based on the corresponding webpage tag data. In fact, in specific embodiments, the process of generating the target cleaning strategy involves selecting the optimal webpage tag data based on the credibility score and container tag filtering to generate the target cleaning strategy.

[0192] For example, based on the webpage tag data exemplified above, the target cleaning strategy that can be generated is:

[0193] {

[0194] "label":"Product Name, Price Cleaning Strategy",

[0195] "key":"roleBox",

[0196] "matches":[

[0197] {

[0198]

[0199]

[0200] Therefore, the web page data crawling method provided in this application solves the problem of excessively high costs associated with relying on manual data generation by introducing a large model and a comprehensive data cleaning strategy. Furthermore, based on the calculation and filtering of credibility scores and container tags, the method ensures the accuracy of data cleaning while improving data cleaning efficiency and guaranteeing a positive user experience during data crawling.

[0201] As an optional embodiment, after cleaning the webpage data to be cleaned using a targeted cleaning strategy to obtain the target crawling data, the process includes:

[0202] Determine whether the target crawled data includes multimodal data other than text data;

[0203] If so, semantic parsing of multimodal data is performed using a specified large model to obtain parsed text; the text data, parsed text, and multimodal data are then structured to obtain the target output data.

[0204] In a specific embodiment, in order to enable the data cleaning process to have multimodal data understanding capabilities, after data cleaning, it is determined whether the currently crawled data includes other multimodal data besides text data, wherein multimodal data includes, but is not limited to, images and videos.

[0205] If multimodal data exists, to facilitate more intuitive data retrieval, a large model is specified to perform semantic parsing on the multimodal data, resulting in parsed text. Furthermore, the text data, parsed text, and multimodal data are structured, and this structured data is used as the final output data presented to the user.

[0206] It should be noted that, similarly, the designated large model can include, but is not limited to, the GPT series, the BERT model, the large model based on the Transformer architecture, LLaMA, and the General Thousand Questions. This application does not limit the selection of the designated large model. In addition, it should be noted that the designated large model in the embodiments of this application and the target large model in the above embodiments can be the same model or different models. The selection can be made according to actual business needs, and this application does not limit this as well.

[0207] In the above embodiments, the method for crawling web page data has been described in detail. This application also provides an embodiment of a web page data crawling device.

[0208] Figure 7 This is a schematic diagram of the structure of a web page data crawling device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes:

[0209] The request parsing module 70 is used to parse the data crawling request input by the user and obtain the parsing result;

[0210] The data crawling module 71 is used to crawl data based on data crawling requests to obtain the web page data to be cleaned.

[0211] The strategy determination module 72 is used to determine, based on the parsing results, whether there is a target cleaning strategy in the pre-built data cleaning strategy library that can be used to clean the web page data to be cleaned; if it exists, the data cleaning module is called; if it does not exist, the strategy generation module is called first, and then the data cleaning module is called.

[0212] Data cleaning module 73 is used to clean the web page data to be cleaned through the target cleaning strategy to obtain the target crawled data;

[0213] The strategy generation module 74 is used to generate a target cleaning strategy based on the target large model according to the web page data to be cleaned.

[0214] Furthermore, the web page data crawling device provided in this application embodiment also includes:

[0215] The first determination module is used to determine the Uniform Resource Locator and request text corresponding to the data crawling request;

[0216] The semantic parsing module is used to perform semantic parsing on the request text in order to obtain keywords in the text;

[0217] The data cleaning dimension determination module is used to determine the data cleaning dimension corresponding to the data crawling request based on the keywords; where the data cleaning dimension is the data type of the keyword.

[0218] The vectorization module is used to vectorize the data request dimensions and Uniform Resource Locators to obtain the target vector; and uses the target vector as the parsing result.

[0219] The webpage structure parsing module is used to parse the webpage structure data to be cleaned and obtain the webpage structure data.

[0220] The DOM structure extraction module is used to extract the DOM structure that conforms to the target cleaning strategy from the web page structure data.

[0221] The DOM node location module is used to locate DOM nodes in the DOM structure that correspond to the data cleaning dimensions.

[0222] The data content extraction module is used to extract the data content corresponding to DOM nodes as the target data for crawling.

[0223] The hierarchy determination module is used to determine the hierarchy of webpage structure tags in the corresponding webpage structure data, and to obtain the positioning tags.

[0224] The data information extraction module is used to extract the data information wrapped by each webpage structure tag based on the positioning tag to obtain the webpage tag data;

[0225] The credibility score determination module is used to determine the credibility score of each webpage tag data through the target large model; the credibility score is used to characterize the credibility of the webpage tag data for obtaining target crawling data. The higher the credibility score, the higher the credibility.

[0226] The target cleaning strategy generation module is used to generate target cleaning strategies based on target webpage tag data; the target webpage tag data consists of data with a credibility score greater than a threshold.

[0227] The first acquisition module is used to acquire the data content and tag attributes of webpage tag data, and determine the data type of the data content;

[0228] The target scoring module is used to determine the semantic score and attribute score of webpage tag data through the target big model; the semantic score is used to characterize the semantic relevance between data content and data type; the attribute score is used to characterize the attribute relevance between tag attributes and data type.

[0229] The weight allocation module is used to assign corresponding weight coefficients to semantic scores and attribute scores.

[0230] The second determining module is used to perform a weighted summation of the semantic score and the attribute score based on the weight coefficients to obtain the credibility score.

[0231] The second acquisition module is used to acquire feedback signals regarding the credibility score;

[0232] The first processing module is used to adjust the weight coefficients and / or increase the number of attribute scores when the feedback signal characterizes the credibility score inaccurately; and to return the steps of determining the semantic score and attribute score of the web page tag data through the target large model; wherein, different attribute scores correspond to different tag attributes; the weight coefficient of the semantic score is greater than the weight coefficient of the attribute score.

[0233] The third acquisition module is used to acquire the target positioning tags of the target webpage tag data;

[0234] The maximum common container label determination module is used to determine the maximum common container label of a specified location label group; the specified location label group consists of target location labels corresponding to different data types;

[0235] The removal module is used to remove specified common container tags from the largest common container tag; the specified common container tag must exist only once in each webpage data to be cleaned.

[0236] The third determining module is used to select the container label located at the innermost layer of the wrapping level among the largest common container labels as the target container label;

[0237] The second processing module is used to generate a target cleaning strategy based on the webpage tag data corresponding to the target container tag.

[0238] The data modality determination module is used to determine whether the target crawled data includes multimodal data other than text data; if so, the data structuring module is called.

[0239] The data structuring module is used to perform semantic parsing on multimodal data using a specified large model to obtain parsed text; and to structure the text data, parsed text, and multimodal data to obtain the target output data.

[0240] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the electronic device includes: a memory 80 for storing computer programs;

[0241] The processor 81 is used to execute computer programs to implement the steps of the web page data crawling method mentioned in the above embodiments.

[0242] The electronic devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0243] The processor 81 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 81 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 81 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 81 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 81 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0244] The memory 80 may include one or more computer-readable storage media, which may be non-transitory. The memory 80 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 80 is used to store at least the following computer program 801, which, after being loaded and executed by the processor 81, is capable of implementing the relevant steps of the web page data crawling method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 80 may also include an operating system 802 and data 803, and the storage method may be temporary or permanent storage. The operating system 802 may include Windows, Unix, Linux, etc. The data 803 may include, but is not limited to, the relevant data involved in the web page data crawling method.

[0245] In some embodiments, the electronic device may further include a display screen 82, an input / output interface 83, a communication interface 84, a power supply 85, and a communication bus 86.

[0246] Those skilled in the art will understand that Figure 8 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0247] The electronic device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the web page data crawling method described in the above embodiments.

[0248] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A method for crawling web page data, characterized in that, The method includes: Parse the user's input data crawling request to obtain the parsing results; Data is crawled based on the data crawling request to obtain the web page data to be cleaned; Based on the analysis results, determine whether there is a target cleaning strategy in the pre-built data cleaning strategy library that can be used to clean the web page data to be cleaned; If it exists, the webpage data to be cleaned is cleaned using the target cleaning strategy to obtain the target crawled data; If it does not exist, the target cleaning strategy is generated based on the target big model according to the web page data to be cleaned, and the step of cleaning the web page data to be cleaned by the target cleaning strategy to obtain the target crawled data is executed.

2. The web page data crawling method as described in claim 1, characterized in that, The parsing of user-inputted data crawling requests yields the following parsing results: Determine the Uniform Resource Locator and request text corresponding to the data crawling request; Perform semantic parsing on the request text to obtain keywords from the text; Based on the keywords, determine the data cleaning dimension corresponding to the data crawling request; wherein, the data cleaning dimension is the data type of the keywords; The data request dimension and the Uniform Resource Locator are vectorized to obtain a target vector; and the target vector is used as the parsing result.

3. The web page data crawling method as described in claim 2, characterized in that, The step of cleaning the webpage data to be cleaned using the target cleaning strategy to obtain the target crawled data includes: The web page data to be cleaned is parsed to obtain web page structure data; Extract the DOM structure that conforms to the target cleaning strategy from the webpage structure data; Locate the DOM node in the DOM structure that corresponds to the data cleaning dimension; Extract the data content corresponding to the DOM node as the target crawling data.

4. The web page data crawling method as described in claim 1, characterized in that, Based on the webpage data to be cleaned, a target cleaning strategy is generated based on the target large model, including: The web page data to be cleaned is parsed to obtain web page structure data; Determine the hierarchical structure of webpage structure tags within the corresponding webpage structure data to obtain the positioning tags; Based on the positioning tags, extract the data information wrapped by each of the webpage structure tags to obtain webpage tag data; The credibility score of each webpage tag data is determined using the target large model; the credibility score is used to characterize the credibility of the webpage tag data in obtaining the target crawled data, and the higher the credibility score, the higher the credibility. The target cleaning strategy is generated based on the target webpage tag data; the target webpage tag data is data whose credibility score is greater than a threshold.

5. The web page data crawling method as described in claim 4, characterized in that, The process of determining the credibility score of each webpage tag data using the target large model includes: Obtain the data content and tag attributes of the webpage tag data, and determine the data type of the data content; Using the target large model, the semantic score and attribute score of the webpage tag data are determined; the semantic score is used to characterize the degree of semantic relevance between the data content and the data type; the attribute score is used to characterize the degree of attribute relevance between the tag attributes and the data type. Assign corresponding weight coefficients to the semantic score and the attribute score; The credibility score is obtained by weighting and summing the semantic score and the attribute score according to the weight coefficients.

6. The web page data crawling method as described in claim 5, characterized in that, After obtaining the credibility score by weighted summation of the semantic score and the attribute score according to the weight coefficients, the process includes: Obtain feedback signals regarding the credibility score; When the feedback signal indicates that the credibility score is inaccurate, the weight coefficient is adjusted and / or the number of attribute scores is increased; and the process returns to the step of determining the semantic score and attribute score of the webpage tag data through the target large model; wherein, different attribute scores correspond to different tag attributes; and the weight coefficient of the semantic score is greater than the weight coefficient of the attribute score.

7. The web page data crawling method as described in claim 5, characterized in that, The step of generating the target cleaning strategy based on the target webpage tag data includes: The target positioning tag for obtaining the target webpage tag data; Determine the largest common container label for a specified location label group; the specified location label group consists of target location labels corresponding to different data types. Remove the specified common container tag from the largest common container tag; the specified common container tag exists only once in each of the web page data to be cleaned. The container label located at the innermost layer of the wrapping hierarchy among the largest common container labels is taken as the target container label; The target cleaning strategy is generated based on the webpage tag data corresponding to the target container tag.

8. The web page data crawling method as described in claim 1, characterized in that, After cleaning the webpage data to be cleaned using the target cleaning strategy to obtain the target crawled data, the process includes: Determine whether the target crawled data includes multimodal data other than text data; If so, semantic parsing is performed on the multimodal data using a specified large model to obtain parsed text; the text data, the parsed text, and the multimodal data are then structured to obtain the target output data.

9. A web page data crawling device, characterized in that, The device includes: The request parsing module is used to parse the data crawling request input by the user and obtain the parsing result; The data crawling module is used to crawl data based on the data crawling request to obtain the web page data to be cleaned; The strategy determination module is used to determine, based on the parsing results, whether there exists a target cleaning strategy in the pre-built data cleaning strategy library that can be used to clean the web page data to be cleaned; if it exists, the data cleaning module is called; if it does not exist, the strategy generation module is called first, and then the data cleaning module is called. The data cleaning module is used to clean the web page data to be cleaned using the target cleaning strategy to obtain the target crawled data. The strategy generation module is used to generate the target cleaning strategy based on the target big model according to the web page data to be cleaned.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the web page data crawling method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the web page data crawling method according to any one of claims 1 to 8.

Images