Webpage data analysis method based on large language model

Abstract

The invention belongs to the technical field of webpage data analysis, and particularly relates to a webpage data analysis method based on a large language model, which comprises the following steps: acquiring a target website URL (Uniform Resource Locator) input by a user; carrying out preliminary analysis on the input URL by utilizing a large language model; the DOM structure of each website is analyzed, and HTML elements and the hierarchical relation thereof are extracted; generating a corresponding data crawling script according to an analysis result; executing the data crawling script, and extracting required data from each website; the extracted data is preprocessed; performing multi-source fusion on the preprocessed data; extracting main information from the fused data; and storing the finally processed and extracted information in a database. According to the method, automatic analysis of complex webpage data, intelligent entity extraction and knowledge expansion and efficient data fusion and duplicate removal are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of web page data analysis, and in particular relates to a web page data analysis method based on a large language model. Background Art

[0002] Web page data parsing technology refers to the process of extracting useful information from web pages. With the development of the Internet, web page data parsing technology is also constantly improving. The following are some common web page data parsing technologies: (1) HTML parser: Use tools such as BeautifulSoup and lxml to parse the DOM structure of the web page and extract the required data from it. This method relies on the structure and tags of the web page. The parsing process is relatively simple, but it is sensitive to changes in the structure of the web page; (2) Regular expressions: Used to match specific patterns or strings in web pages to extract relevant data. It has flexibility and powerful text processing capabilities, but it is difficult to write and maintain complex regular expressions; (3) XPath and CSS selectors: Able to locate elements in web pages to extract data. This method is intuitive and easy to use, but it also relies on the structure of the web page. Changes in the structure of the web page may cause parsing failure; (4) Template matching-based methods: Use predefined templates to match web page content and extract data from it. It is suitable for web pages with fixed structures, but it is not effective for dynamically changing web pages.

[0003] Although existing web data parsing technologies have met the needs of data extraction to a certain extent, there are still some obvious problems and shortcomings: (1) Strong dependence on web page structure: Existing technologies usually rely on the DOM structure of web pages. Changes in web page structure (such as website revisions) will cause parsing rules to become invalid, requiring frequent adjustments to parsing rules and high maintenance costs; (2) Insufficient data knowledge supplementation: In the data parsing process, usually only attention is paid to data extraction, while the supplementation and understanding of data background knowledge are ignored. For example, the parsed data may lack context or relevant knowledge, resulting in reduced data utilization value; (3) Difficulty in fusing multiple data sources: When processing data from multiple data sources, existing technologies lack effective data fusion methods. The data formats, structures, and semantics of different data sources may differ, and existing technologies are difficult to automatically align and fuse data, resulting in data redundancy and inconsistency problems; (4) Insufficient processing capabilities for dynamic and complex web pages: When processing dynamically generated content (such as content rendered by JavaScript) and complex web pages, the processing logic is usually complex and scalability is poor. Data extraction from these web pages requires more advanced parsing technology and dynamic content processing capabilities; (5) Lack of intelligence and automation: The formulation and adjustment of parsing rules rely on manual intervention and experience, and lack of intelligence and automation capabilities, resulting in low efficiency and difficulty in coping with large-scale and diverse web page data parsing needs. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to address the deficiencies of the prior art and provide a web page data parsing method based on a large language model that automatically adapts to changes in web page structure and realizes intelligent entity extraction and knowledge expansion.

[0005] Technical solution: The web page data parsing method based on a large language model of the present invention comprises the following steps: S1: Obtain target information keywords and several target website URLs input by the user; S2: Use the large language model to perform preliminary analysis on the input URL and identify the website structure; S3: Detailed analysis of the DOM structure of each website to extract the HTML elements and their hierarchical relationships of each website; S4: Generate corresponding data crawling scripts based on the parsing results of each website; S5: Execute the data crawling script to extract data related to the target information keywords from each website; S6: preprocessing the extracted data; S7: Multi-source fusion of pre-processed data to integrate data from different websites; S8: Calling the term search engine to automatically search and supplement relevant information based on the key entity vocabulary; S9: extracting information related to target information keywords from the fused and supplemented data; S10: Storing the extracted information in a database.

[0006] To further improve the above technical solution, the S2 uses a large language model to perform a preliminary analysis of the input URL, including: analyzing the segmentation tags in each website homepage and category page and link tags , identify the category page and detail page of the target information.

[0007] Furthermore, S3 includes: link tags extracted from the website homepage and category pages Get the URL of the detail page; use the semantic understanding ability of the large language model to filter the tags and text in the detail page, summarize the information hierarchy of the detail page and identify the HTML tags and paths where the target information is located.

[0008] Furthermore, S4 includes: the large language model automatically generates a data crawler script based on the obtained HTML elements and their hierarchical relationships, and autonomously adjusts the parsing logic of different websites. The data crawler script is used to traverse the classification page of each website, enter the detailed page, and extract the target information. The large language model automatically generates a crawler script based on demand, converts the specific website parsing logic, can adapt to changes in website structure, reduce manual continuous maintenance, and has stronger flexibility and adaptability.

[0009] Furthermore, S5 includes: when crawling web pages, the large language model summarizes the paging mechanism by identifying the changes in URL parameters in the link tags, realizes automatic paging and crawls all detail page links; the large language model calls the dependent library to simulate browser behavior, handles dynamic loading problems, and ensures data integrity.

[0010] Furthermore, S6 includes: the large language model determines the complexity of the web page structure according to the identified number of nested tag layers; if the number of nested layers is 3 or less, regular expressions and HTML parsing libraries are used to clean up HTML tags; if the number of nested layers is greater than 3, HTML tags are processed by a pre-trained NLP model; the large language model automatically detects the encoding format of the website through the HTTP header information and text content of the web page, reads and decodes the data, converts it into the target encoding format, and accurately identifies and filters noise data.

[0011] Furthermore, S7 includes: a large language model identifies and unifies the names of the same entity in different websites, merges parameters provided by different websites, integrates pictures provided by different websites, removes duplicate information, and integrates data from different websites.

[0012] Furthermore, invoking the entry search engine to automatically search and supplement relevant information in S8 includes: generating a search query, extracting and supplementing background information and historical data of the target information.

[0013] Furthermore, S9 includes: using a large language model to parse the fused data, identifying fields related to the target information, and performing structured storage.

[0014] Furthermore, the S10 includes: generating a database storage format, and storing data in various formats such as texts and pictures in the database.

[0015] Beneficial effects: Compared with the prior art, the present invention achieves the following technical effects by introducing a large language model (such as a large-scale language model) and utilizing its powerful natural language understanding and generation capabilities: 1. Automatically adapt to changes in web page structure The present invention can automatically adapt to changes in web page structure and reduce reliance on manual parsing rules. Existing parsing technology is overly dependent on the DOM structure of web pages. Changes in web page structure (such as website revisions) can cause parsing rules to become invalid, resulting in high maintenance costs. The present invention, through the application of a large language model, can accurately parse complex web page structures, automatically handle HTML tags and encoding issues, and effectively remove noise data, ensuring the high quality and consistency of captured data and improving the stability and robustness of parsing.

[0016] 2. Intelligent entity extraction and knowledge expansion By utilizing the multimodal and cross-domain understanding capabilities of large language models, the present invention performs topic analysis and keyword entity extraction on captured text or multimodal data, and combines Internet entry information and the latest public opinion data to achieve intelligent expansion of data and knowledge supplementation, providing more comprehensive and in-depth data insights.

[0017] 3. Efficient data fusion and deduplication The present invention utilizes a large language model to restructure heterogeneous data and reorganize and generate data storage structures, thereby minimizing data redundancy and improving data storage efficiency. It can merge terms with the same semantics and efficiently remove duplicate information, greatly improving data storage efficiency and consistency and reducing data redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0020] Example 1: Figure 1 The web page data parsing method based on the large language model shown includes the following steps: S1: Obtain target information keywords and several target website URLs input by the user; S2: Use the large language model to perform preliminary analysis on the input URL and identify the website structure; S3: Detailed analysis of the DOM structure of each website to extract the HTML elements and their hierarchical relationships of each website; S4: Generate corresponding data crawling scripts based on the parsing results of each website; S5: Execute the data crawling script to extract data related to the target information keywords from each website; S6: preprocessing the extracted data; S7: Multi-source fusion of pre-processed data to integrate data from different websites; S8: Calling the term search engine to automatically search and supplement relevant information based on the key entity vocabulary; S9: extracting information related to target information keywords from the fused and supplemented data; S10: Storing the extracted information in a database.

[0021] The present invention aims to construct an intelligent agent that integrates data request and analysis, entity extraction and extended knowledge supplement, and multi-source analysis data fusion.

[0022] 1. Accurate analysis of web page data and data structure design Use large language models (GPT4, GLM4, etc.) to analyze and deconstruct the hierarchical structure of the website and the information elements of each sub-page, design a robust and scalable crawling script on the website, and pre-process the data while crawling it, including combing HTML tags, removing noise data, and dealing with encoding issues.

[0023] 2. Entity extraction and expanded knowledge supplement Use large language models to perform topic analysis and keyword entity extraction on captured text or multimodal data, retrieve Internet entry information of keywords and the latest public opinion information, and perform corresponding analysis and mapping.

[0024] 3. Multi-source analysis data fusion The data captured from multiple websites may have inconsistent data storage structures (JSON structures), inconsistent nouns with the same semantics, and duplicate data information. The large language model is used to reorganize and generate the data storage structure to minimize data redundancy and improve data storage efficiency; the semantics of the terms are judged and the terms with the same semantics are merged; and duplicate information is efficiently deduplicated and merged.

[0025] Example 2: Take the implementation case of AK47 information crawling and fusion as an example: (1) Website input The user enters the URLs of multiple military equipment websites in the website input module, for example: Military website 1: https: / / www.militaryfactory.com / Military website 2: https: / / www.militarytoday.com / Military website 3: http: / / pewpewpew.work / index.htm (2) Preliminary analysis Task: Perform preliminary analysis on the input URL and identify the main structure and content of the website.

[0026] Specific operations: Use a large language model (such as GPT-4) to analyze the homepage and category pages of each website, and identify the category pages of military equipment and the detailed pages of specific equipment.

[0027] (3) Structural analysis Task: Analyze the DOM structure of each website in detail and extract each HTML element and its hierarchical relationship.

[0028] Specific operations: The large language model parses the HTML structure of each page and identifies the HTML tags and paths where key information such as equipment names, pictures, and performance parameters are located.

[0029] (4) Generate crawling script Task: Generate corresponding data crawling scripts based on the analysis results of each website.

[0030] Specific operations: The large language model generates a Python crawler script, which is responsible for traversing the military equipment classification page of each website, entering the detailed page of each specific equipment, and extracting the equipment's name, picture, performance parameters and other information.

[0031] (5) Data crawling Task: Execute the data scraping script to extract the required AK47 automatic rifle information from each website.

[0032] Specific operation: The large language model handles issues such as paging and dynamic loading during the crawling process to ensure data integrity. For example, extract information from the AK47 page of multiple military websites.

[0033] (6) Preprocess the extracted data Task: Perform pre-processing operations on the crawled data, such as cleaning HTML tags, removing noise data, and dealing with encoding problems.

[0034] Specific operations: Clean HTML tags: The large language model uses regular expressions or HTML parsing libraries (such as BeautifulSoup) to remove irrelevant HTML tags and retain useful information.

[0035] Removing noisy data: The large language model uses pre-defined rules or machine learning models to identify and filter out irrelevant information such as advertisements and navigation bars.

[0036] Coding problem handling: The large language model detects and converts the encoding formats of different websites to ensure the correct display of data.

[0037] (7) Multi-source fusion of preprocessed data Task: Perform multi-source fusion on the preprocessed data and integrate data from different websites.

[0038] Specific operations: Name unification: The large language model identifies and unifies the names of AK47 automatic rifles on different websites, such as "ak-47" and "AK47"; parameter merging: The large language model merges the performance parameters (such as speed, range, weapon configuration, etc.) provided by different websites, and takes the average value or the most reliable data; image integration: The large language model integrates the AK47 automatic rifle images provided by different websites and removes duplicate images.

[0039] (8) Supplementary information Task: Call the entry search engine (such as Wiki, Baidu, Google, etc.), automatically search for relevant information based on the key entity vocabulary (such as "ak47") and supplement it.

[0040] Specific operations: The large language model generates search queries, extracts and supplements information such as the historical background, research and development process, and countries of use of the AK47 automatic rifle.

[0041] (9) Information extraction Task: Extract the main AK47 automatic rifle information from the fused data.

[0042] Specific operations: Use a large language model to deeply analyze the fused data and identify the main information fields of the AK47 automatic rifle.

[0043] Extract key information: The large language model extracts key information such as the name, model, performance parameters, and country of use of the AK47, and stores it in a structured manner.

[0044] (10) Data storage Task: Store the final processed and extracted information into a MongoDB database.

[0045] Specific operations: The large language model generates the MongoDB storage format and stores data in various formats including text, pictures, audio, and video in the database to facilitate subsequent query and analysis.

[0046] MongoDB storage example: { "name": "ak-47", "model": "automatic rifle", "manufacturer": "Russian", "year": "1947", "weight": "3.47kg", "length": "880mm", "cartridge":"7.62×39mm", "rate_of_fire": "600 rounds / min", "images": [ "https: / / military website1 / images / ak47_1.jpg", "https: / / military website 2 / images / ak47_2.jpg" ] } As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A web page data parsing method based on a large language model, characterized in that: The steps include: S1: Obtain target information keywords and several target website URLs input by the user; S2: Use the large language model to perform preliminary analysis on the input URL and identify the website structure; S3: Detailed analysis of the DOM structure of each website to extract the HTML elements and their hierarchical relationships of each website; S4: Generate corresponding data crawling scripts based on the parsing results of each website; S5: Execute the data crawling script to extract data related to the target information keywords from each website; S6: preprocessing the extracted data; S7: Multi-source fusion of pre-processed data to integrate data from different websites; S8: Calling the term search engine to automatically search and supplement relevant information based on the key entity vocabulary; S9: extracting information related to target information keywords from the fused and supplemented data; S10: Storing the extracted information in a database.

2. The web page data parsing method based on a large language model according to claim 1, characterized in that: The S2 uses a large language model to perform a preliminary analysis of the input URL, including: analyzing the segmentation tags in each website homepage and category page and link tags , identify the category page and detail page of the target information.

3. The web page data parsing method based on a large language model according to claim 2, characterized in that: The S3 includes: link tags extracted from the website homepage and category pages Get the URL of the detail page; use the semantic understanding ability of the large language model to filter the tags and text in the detail page, summarize the information hierarchy of the detail page and identify the HTML tags and paths where the target information is located.

4. The web page data parsing method based on a large language model according to claim 1, characterized in that: The S4 includes: the large language model automatically generates a data crawler script according to the obtained HTML elements and their hierarchical relationships, and autonomously adjusts the parsing logic of different websites. The data crawler script is used to traverse the classification page of each website, enter the detailed page, and extract the target information.

5. The web page data parsing method based on a large language model according to claim 1, characterized in that: The S5 includes: when crawling web pages, the large language model summarizes the paging mechanism by identifying the changes in URL parameters in the link tags, realizes automatic paging and crawls all detail page links; the large language model calls the dependent library to simulate the browser behavior and handles the dynamic loading problem.

6. The web page data parsing method based on a large language model according to claim 1, characterized in that: The S6 includes: the large language model determines the complexity of the web page structure according to the identified number of nested tags. If the number of nested layers is 3 or less, regular expressions and HTML parsing library are used to clean up HTML tags. If the number of nested layers is greater than 3, the HTML tags are processed by a pre-trained NLP model. The large language model automatically detects the encoding format of the website through the HTTP header information and text content of the web page, reads and decodes the data, and converts it into the target encoding format.

7. The web page data parsing method based on a large language model according to claim 1, characterized in that: The S7 includes: a large language model identifies and unifies the names of the same entity in different websites, merges the parameters provided by different websites, integrates the pictures provided by different websites, removes duplicate information, and integrates data from different websites.

8. The web page data parsing method based on a large language model according to claim 1, characterized in that: The calling of the term search engine to automatically search and supplement relevant information in S8 includes: generating a search query, extracting and supplementing background information and historical data of the target information.

9. The web page data parsing method based on a large language model according to claim 1, characterized in that: The S9 includes: using a large language model to parse the fused data, identifying fields related to the target information, and performing structured storage.

10. The web page data parsing method based on a large language model according to claim 1, characterized in that: The S10 includes: generating a database storage format and storing it in a database.

Images