A Web Page Data Parsing Method Based on a Large Language Model

Abstract

This invention belongs to the field of web page data parsing technology, specifically relating to a web page data parsing method based on a large language model. The method includes: obtaining the target website URL input by the user; performing preliminary parsing of the input URL using a large language model; parsing the DOM structure of each website and extracting HTML elements and their hierarchical relationships; generating corresponding data crawling scripts based on the parsing results; executing the data crawling scripts to extract the required data from each website; preprocessing the extracted data; fusing the preprocessed data from multiple sources; extracting key information from the fused data; and storing the finally processed and extracted information in a database. This invention achieves automatic parsing of complex web page data, intelligent entity extraction and knowledge expansion, as well as efficient data fusion and deduplication.

Description

Technical Field

[0001] This invention belongs to the field of web page data parsing technology, specifically relating to a web page data parsing method based on a large language model. Background Technology

[0002] Web page data parsing technology refers to the process of extracting useful information from web pages. With the development of the Internet, web page data parsing technology is also constantly improving. Here are some common web page data parsing technologies: (1) HTML parser: Using tools such as BeautifulSoup and lxml to parse the DOM structure of the web page and extract the required data. This method depends on the structure and tags of the web page. The parsing process is relatively simple, but it is sensitive to changes in the web page structure. (2) Regular expression: Used to match specific patterns or strings in the web page and extract relevant data. It has flexibility and powerful text processing capabilities, but it is difficult to write and maintain complex regular expressions. (3) XPath and CSS selectors: Able to locate elements in the web page to extract data. This method is intuitive and easy to use, but it also depends on the structure of the web page. Changes in the web page structure may lead to parsing failure. (4) Template matching method: Uses predefined templates to match web page content and extract data. It is suitable for web pages with fixed structures, but it is not effective for dynamically changing web pages.

[0003] Although existing web page data parsing technologies meet the data extraction needs to a certain extent, there are still some obvious problems and shortcomings: (1) Strong dependence on web page structure: Existing technologies usually rely on the DOM structure of web pages. Changes in the web page structure (such as website redesign) will cause the parsing rules to fail, requiring frequent adjustments to the parsing rules and resulting in high maintenance costs; (2) Insufficient data knowledge supplementation: In the data parsing process, only the extraction of data is usually focused on, while the supplementation and understanding of the data background knowledge is ignored. For example, the parsed data may lack context or relevant knowledge, which reduces the utilization value of the data; (3) Difficulty in merging multiple data sources: When processing data from multiple data sources, existing technologies lack effective data fusion methods. The data formats, structures and semantics of different data sources may differ, and existing technologies have difficulty automating data alignment and fusion, leading to data redundancy and inconsistency problems; (4) Insufficient ability to process dynamic and complex web pages: When processing dynamically generated content (such as content rendered by JavaScript) and complex web pages, the processing logic is usually complex and the scalability is poor. Data extraction from these web pages requires more advanced parsing techniques and dynamic content processing capabilities; (5) Lack of intelligence and automation: In the formulation and adjustment of parsing rules, it relies on manual intervention and experience, lacks intelligence and automation capabilities, resulting in low efficiency and difficulty in coping with large-scale and diverse web page data parsing needs. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies by providing a web page data parsing method based on a large language model, which automatically adapts to changes in web page structure and achieves intelligent entity extraction and knowledge expansion.

[0005] Technical solution: The webpage data parsing method based on a large language model described in this invention includes the following steps:

[0006] S1: Obtain the target information keywords and several target website URLs input by the user;

[0007] S2: Use a large language model to perform preliminary parsing of the input URL and identify the website structure;

[0008] S3: Perform detailed analysis of the DOM structure of each website, extracting the HTML elements and their hierarchical relationships;

[0009] S4: Generate corresponding data crawling scripts based on the parsing results of each website;

[0010] S5: Execute the data crawling script to extract data related to the target information keywords from each website;

[0011] S6: Preprocess the extracted data;

[0012] S7: Perform multi-source fusion on the preprocessed data, integrating data from different websites;

[0013] S8: Call the term search engine to automatically search and supplement relevant information based on the key entity term list;

[0014] S9: Extract information related to the target information keywords from the fused and supplemented data;

[0015] S10: Store the extracted information in the database.

[0016] To further improve the above technical solution, the preliminary parsing of the input URL using a large language model in S2 includes: analyzing the segmentation tags in the homepage and category pages of each website. and link tags It identifies the category page and detail page of the target information.

[0017] Furthermore, S3 includes: link tags extracted from the website homepage and category pages. The URL of the details page is obtained; the semantic understanding capability of the large language model is used to filter the tags and text in the details page, summarize the hierarchical relationship of the information on the details page, and identify the HTML tags and paths where the target information is located.

[0018] Furthermore, S4 includes: the large language model automatically generates a data crawler script based on the obtained HTML elements and their hierarchical relationships, and autonomously adjusts the parsing logic for different websites. The data crawler script is used to traverse the category pages of each website, enter the detailed pages, and extract target information. The large language model automatically generates crawler scripts according to needs and transforms specific website parsing logic, which can adapt to changes in website structure, reduce continuous manual maintenance, and has greater flexibility and adaptability.

[0019] Furthermore, S5 includes: when crawling web pages, the large language model summarizes the pagination mechanism by identifying changes in URL parameters in link tags, thereby achieving automatic pagination and crawling all detail page links; the large language model calls dependent libraries to simulate browser behavior, handle dynamic loading issues, and ensure data integrity.

[0020] Furthermore, S6 includes: the large language model determines the complexity of the webpage structure based on the number of nested tags identified; if the nesting level is 3 or less, regular expressions and an HTML parsing library are used to clean up HTML tags; if the nesting level is greater than 3, a pre-trained NLP model is used to process HTML tags; the large language model automatically detects the website's encoding format through the webpage's HTTP header information and text content, reads and decodes the data, converts it to the target encoding format, and accurately identifies and filters noisy data.

[0021] Furthermore, S7 includes: using a large language model to identify and unify the names of the same entity in different websites, merging parameters provided by different websites, integrating images provided by different websites, and removing duplicate information to integrate data from different websites.

[0022] Furthermore, the step of calling the term search engine to automatically search for and supplement relevant information in S8 includes: generating a search query, and extracting and supplementing background information and historical data of the target information.

[0023] Furthermore, S9 includes: using a large language model to parse the fused data, identifying fields related to the target information, and storing them in a structured manner.

[0024] Furthermore, S10 includes: generating a database storage format and storing data in various formats such as text and images into the database.

[0025] Beneficial effects: Compared with existing technologies, this invention achieves the following technical effects by introducing a large language model (such as a large-scale language model) and leveraging its powerful natural language understanding and generation capabilities:

[0026] 1. Automatically adapts to changes in webpage structure

[0027] This invention can automatically adapt to changes in webpage structure, reducing reliance on manual parsing rules. Existing parsing technologies rely excessively on the DOM structure of webpages; changes in the webpage structure (such as website redesign) can cause parsing rules to become ineffective, resulting in high maintenance costs. This invention, through the application of a large language model, can accurately parse complex webpage structures, automatically handle HTML tags and encoding issues, and effectively remove noisy data, ensuring high-quality and consistent crawled data, and improving the stability and robustness of parsing.

[0028] 2. Intelligent entity extraction and knowledge expansion

[0029] Leveraging the multimodal and cross-domain understanding capabilities of large language models, this invention performs topic analysis and keyword entity extraction on captured text or multimodal data. By combining internet terminology information and the latest public opinion data, it achieves intelligent data expansion and knowledge supplementation, providing more comprehensive and in-depth data insights.

[0030] 3. Efficient data fusion and deduplication

[0031] This invention utilizes a large language model to restructure heterogeneous data, generating a restructured data storage structure that minimizes data redundancy and improves data storage efficiency. It can merge semantically similar terms and efficiently remove duplicate information, greatly improving data storage efficiency and consistency while reducing data redundancy. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0034] Example 1: As Figure 1 The webpage data parsing method based on a large language model, as shown, includes the following steps:

[0035] S1: Obtain the target information keywords and several target website URLs input by the user;

[0036] S2: Use a large language model to perform preliminary parsing of the input URL and identify the website structure;

[0037] S3: Perform detailed analysis of the DOM structure of each website, extracting the HTML elements and their hierarchical relationships;

[0038] S4: Generate corresponding data crawling scripts based on the parsing results of each website;

[0039] S5: Execute the data crawling script to extract data related to the target information keywords from each website;

[0040] S6: Preprocess the extracted data;

[0041] S7: Perform multi-source fusion on the preprocessed data, integrating data from different websites;

[0042] S8: Call the term search engine to automatically search and supplement relevant information based on the key entity term list;

[0043] S9: Extract information related to the target information keywords from the fused and supplemented data;

[0044] S10: Store the extracted information in the database.

[0045] The present invention aims to construct an intelligent agent that integrates data request and parsing, entity extraction and knowledge expansion, and multi-source data fusion.

[0046] 1. Precise parsing and data structure design of web page data

[0047] By using large language models (GPT4, GLM4, etc.) to analyze and deconstruct the hierarchical structure of the website and the information elements of each subpage, a robust and scalable crawling script is designed for the website. While crawling data, the script preprocesses the data, including sorting out HTML tags, removing noisy data, and handling encoding issues.

[0048] 2. Entity Extraction and Knowledge Expansion Supplement

[0049] Large language models are used to perform topic analysis and keyword entity extraction on the captured text or multimodal data, and to retrieve internet term information and the latest public opinion information for the keywords, and perform corresponding parsing and mapping.

[0050] 3. Multi-source data fusion

[0051] Data scraped from multiple websites may have issues such as inconsistent data storage structures (JSON structure), inconsistent semantic terms, and duplicate data information. By using a large language model to reorganize and generate data storage structures, data redundancy can be minimized and data storage efficiency can be improved. Semantic judgment of entries is performed, and entries with the same semantic meaning are merged. Duplicate information is efficiently deduplicated and merged.

[0052] Example 2: Implementation case of AK47 information crawling and fusion:

[0053] (1) Website input

[0054] Users can enter the URLs of multiple equipment-related websites in the website input module;

[0055] (2) Preliminary analysis

[0056] Task: Perform preliminary parsing of the input URL to identify the main structure and content of the website.

[0057] Specific operations: Use a large language model (such as GPT-4) to analyze the homepage and category pages of each website, and identify the category pages of military equipment and the detailed pages of specific equipment.

[0058] (3) Structural analysis

[0059] Task: Perform a detailed analysis of the DOM structure of each website, extracting each HTML element and its hierarchical relationship.

[0060] Specific operation: The large language model parses the HTML structure of each page and identifies the HTML tags and paths containing key information such as equipment name, image, and performance parameters.

[0061] (4) Generate crawling script

[0062] Task: Generate corresponding data crawling scripts based on the parsing results of each website.

[0063] Specific operations: The large language model generates a Python crawler script, which is responsible for traversing the military equipment category pages of each website, entering the detailed pages of each specific piece of equipment, and extracting information such as the equipment's name, image, and performance parameters.

[0064] (5) Data crawling

[0065] Task: Execute a data scraping script to extract the required AK47 assault rifle information from each website.

[0066] Specific operations: The large language model handles issues such as pagination and dynamic loading during the crawling process to ensure data integrity. For example, extracting information from AK47 pages on multiple military websites.

[0067] (6) Preprocess the extracted data

[0068] Task: Perform preprocessing operations on the crawled data, such as cleaning HTML tags, removing noisy data, and handling encoding issues.

[0069] Specific steps: Clean up HTML tags: The large language model uses regular expressions or HTML parsing libraries (such as BeautifulSoup) to remove irrelevant HTML tags and retain useful information.

[0070] Noise removal: Large language models identify and filter out irrelevant information such as advertisements and navigation bars through predefined rules or machine learning models.

[0071] Encoding issues: The large language model detects and converts the encoding formats of different websites to ensure the correct display of data.

[0072] (7) Perform multi-source fusion on the preprocessed data.

[0073] Task: To perform multi-source fusion on the preprocessed data, integrating data from different websites.

[0074] Specific operations include: Name unification: The large language model identifies and unifies the names of AK47 assault rifles from different websites, such as "ak-47" and "AK47"; Parameter merging: The large language model merges the performance parameters (such as speed, range, weapon configuration, etc.) provided by different websites and takes their average value or the most reliable data; Image integration: The large language model integrates the AK47 assault rifle images provided by different websites and removes duplicate images.

[0075] (8) Information Supplement

[0076] Task: Call a term search engine (such as Wiki, Baidu, Google, etc.) to automatically search for relevant information and supplement it based on a key entity thesaurus (such as "ak47").

[0077] Specific operations: The large language model generates search queries, extracts and supplements information such as the historical background, development process, and countries of use of the AK47 automatic rifle.

[0078] (9) Information extraction

[0079] Task: Extract key AK47 assault rifle information from the fused data.

[0080] Specific operation: Use a large language model to perform deep analysis on the fused data to identify the main information fields of the AK47 automatic rifle.

[0081] Extracting key information: The large language model extracts key information such as the name, model, performance parameters, and country of use of the AK47, and stores it in a structured manner.

[0082] (10) Data storage

[0083] Task: Store the final processed and extracted information into a MongoDB database.

[0084] Specific operations: The large language model generates MongoDB storage format, storing data in the database in various formats, including text, images, audio, and video, for easy subsequent querying and analysis.

[0085] MongoDB storage example:

[0086] {

[0087] "name": "ak-47",

[0088] "model": "automatic rifle",

[0089] "manufacturer": "Russian",

[0090] "year": "1947",

[0091] "weight": "3.47kg",

[0092] "length": "880mm",

[0093] "cartridge":"7.62×39mm",

[0094] "rate_of_fire": "600 rounds / min",

[0095] }

[0096] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A webpage data parsing method based on a large language model, characterized in that, Includes the following steps: S1: Obtain the target information keywords and several target website URLs input by the user; S2: Use a large language model to perform preliminary parsing of the input URL and identify the website structure; S3: Perform detailed analysis of the DOM structure of each website, extracting the HTML elements and their hierarchical relationships; S4: The large language model automatically generates a data crawler script based on the HTML elements and their hierarchical relationships. The data crawler script is used to traverse the category pages of each website, enter the detailed pages, extract target information, and make autonomous adjustments to the parsing logic of different websites so that the data crawling script can adapt to changes in the website structure. S5: Execute the data crawling script to extract data related to the target information keywords from each website, including: when crawling web pages, the large language model summarizes the pagination mechanism by identifying changes in URL parameters in link tags, realizes automatic pagination and crawls all detail page links, and the large language model calls the dependent library to simulate browser behavior and handle dynamic loading issues; S6: Preprocess the extracted data; S7: Perform multi-source fusion on the preprocessed data to integrate data from different websites, including: the large language model identifies and unifies the name of the same entity in different websites, merges the parameters provided by different websites, integrates the images provided by different websites, and removes duplicate information to integrate data from different websites; S8: Call the term search engine to automatically search and supplement relevant information based on the key entity term list; the key entity term list is determined by the large language model from the target information keywords; S9: Extract information related to the target information keywords from the fused and supplemented data; S10: Store the extracted information in the database.

2. The webpage data parsing method based on a large language model according to claim 1, characterized in that, The initial parsing of the input URL using a large language model in S2 includes: analyzing the segmentation tags in each website's homepage and category pages. and link tags It identifies the category page and detail page of the target information.

3. The webpage data parsing method based on a large language model according to claim 2, characterized in that, S3 includes: link tags extracted from the website homepage and category pages. The URL of the details page is obtained; the semantic understanding capability of the large language model is used to filter the tags and text in the details page, summarize the hierarchical relationship of the information on the details page, and identify the HTML tags and paths where the target information is located.

4. The webpage data parsing method based on a large language model according to claim 1, characterized in that, S6 includes: the large language model determines the complexity of the webpage structure based on the number of nested tags identified. If the number of nested layers is 3 or less, regular expressions and an HTML parsing library are used to clean up HTML tags. If the number of nested layers is greater than 3, a pre-trained NLP model is used to process HTML tags. The large language model automatically detects the website's encoding format through the webpage's HTTP header information and text content, reads and decodes the data, and converts it into the target encoding format.

5. The webpage data parsing method based on a large language model according to claim 1, characterized in that, The S8 step of calling the term search engine to automatically search for and supplement relevant information includes: generating search queries, extracting and supplementing background information and historical data of the target information.

6. The webpage data parsing method based on a large language model according to claim 1, characterized in that, The S9 includes: using a large language model to parse the fused data, identifying fields related to the target information, and storing them in a structured manner.

7. The webpage data parsing method based on a large language model according to claim 1, characterized in that, S10 includes: generating a database storage format and storing it in the database.

Images