Webpage expert information intelligent collection and structuring method based on semantic understanding

Abstract

The application discloses a webpage expert information intelligent collection and structuring method based on semantic understanding, relates to the technical field of data collection, natural language processing and information extraction, and is characterized in that: a browser plug-in is used to assist a business staff in selecting sample elements, all similar expert items on a page are automatically matched, a multi-dimensional matching mechanism integrating a DOM structure, a CSS selector, visual features and semantic analysis is adopted, an expert list is accurately identified and a detail page link is extracted, page content is then batch acquired, after preprocessing, the page content is input into a large language model to extract expert fields according to structured prompt words, and finally, the expert fields are mapped, normalized, quality checked, synchronized to an expert database and a collection report is generated; the webpage expert information intelligent collection and structuring method based on semantic understanding is used to automatically and efficiently extract expert data from expert information pages scattered on the Internet and synchronize the expert data to an expert database.

Description

Technical Field

[0001] This invention relates to the fields of data acquisition, natural language processing and information extraction, and in particular to a method for intelligent acquisition and structuring of web page expert information based on semantic understanding. Background Technology

[0002] Review service companies typically maintain a large expert database to match qualified experts with various review projects. One source of this expert data is publicly available expert information pages on platforms such as universities, research institutions, academic societies, and talent programs on the internet.

[0003] Traditional methods of collecting expert data mainly rely on manual operation: business personnel find the expert list page through search engines or specific websites, click to enter the details page of each expert one by one, manually copy information such as name, title, research direction, work unit, contact information, academic achievements, etc., paste the information into the company's internal system, format it, and then enter it into the database.

[0004] This approach suffers from problems such as low efficiency, poor consistency, high cost, limited coverage, and delayed updates. Each expert's information needs to be manually accessed and entered one by one, resulting in slow data accumulation. Different personnel have different input formats, which can easily lead to dirty data. It requires a large amount of human resources for repetitive work, and it is difficult for manual labor to cover the massive amount of website and expert information. After changes in expert information, it is difficult to update in a timely manner.

[0005] While existing web crawling technologies can automate the capture of web page data, they suffer from significant limitations in practical applications, such as large differences in web page structure, high maintenance costs, weak anti-interference capabilities, and poor support for dynamic rendering. The expert lists and detail pages of different websites have different structures, requiring the writing of separate crawling rules or XPath / CSS selectors for each website. After the target website is redesigned, the original crawling rules become invalid, requiring continuous maintenance. Web pages often contain noisy information such as advertisements, navigation bars, recommended content, and comment sections, which can easily interfere with the extraction of the main text. Many modern websites use JavaScript to dynamically load content, making it difficult for traditional crawlers to directly obtain complete data. Summary of the Invention

[0006] The purpose of this invention is to provide a method for intelligent collection and structuring of web page expert information based on semantic understanding. By enabling interactive element selection between the user and the web page through a browser plugin, and combining the semantic understanding capabilities of DOM structure analysis, visual feature matching, and large language models, the method automatically discovers all expert detail page links from the expert list page, accesses the detail pages in batches and extracts structured expert information, and finally synchronizes it to the expert database, thereby significantly improving the efficiency, accuracy, and maintainability of expert data collection.

[0007] To achieve the above objectives, this invention provides a method for intelligent collection and structuring of webpage expert information based on semantic understanding, comprising the following steps: S1. The browser plugin is installed in the browser of the business personnel. When the business personnel visit a website containing expert information, the plugin obtains the access permission of the current page through the browser extension API and listens for the page load event. S2. On the expert list page, business personnel can manually select any list item representing an expert through the interactive interface provided by the browser plugin. S3, the plugin, is based on the sample elements selected by the S2 user and automatically matches all expert elements with similar structures and functions in the page DOM tree; S4. For each expert element identified in S3, the plugin automatically extracts the hyperlinks, deduplicates and filters all extracted expert detail page links, and generates a list of detail page URLs to be accessed. S5. The plugin sends the list of detail page URLs to the cloud server or allows the plugin to open each detail page in batches in the browser according to the access policy. S6. Obtain the complete HTML content of each expert's details page and preprocess the obtained HTML; S7. Send the preprocessed web page text content to the large language model in the cloud, and guide the model to recognize and extract expert information fields through structured prompt words; S8. Receive JSON data returned by LLM and perform field mapping and format standardization according to the company's internal expert data model; S9. Perform quality verification on the expert data after S8 structuring, and provide processing strategies for conflicting data; S10. Synchronously write the verified expert data into the company's expert database and generate a collection report.

[0008] Preferably, in S1, for websites that use JavaScript dynamic rendering, the plugin will wait for the main elements of the page to finish loading to ensure that the content of the expert list on the page is fully presented.

[0009] Preferably, the specific content of S2 is as follows: The plugin records information about the element selected by the user, including the DOM path, CSS selector, element tag, class name, ID, the element's relative position, size, and visual style characteristics on the page, the text content inside the element and its semantic characteristics, and the structural characteristics of adjacent sibling elements and parent container elements.

[0010] Preferably, the specific content of S3 is as follows: S31. Extract the DOM tree structure features of the sample elements, search for other elements with similar DOM structures in the page, and calculate the structural similarity score between the candidate elements and the sample elements. S32. Based on the class, data-*, and role attributes of the sample elements, generate reusable CSS selectors, match the set of elements on the page that match the same or similar selectors, and calculate the CSS attribute matching score. S33. Extract the visual features of the sample elements, group elements with similar visual features into one category, and calculate the visual similarity score. S34. Perform semantic analysis on the text in the sample elements and their sub-elements, identify whether they contain semantic features, filter out elements that do not match the semantics, and calculate the semantic similarity score. S35. Based on the scores from multiple dimensions including structural similarity, CSS attribute matching, visual similarity, and semantic matching from S31 to S34, the candidate elements are ranked, and elements with a confidence level higher than the threshold are selected as expert list items.

[0011] Preferably, in S4, if the expert element itself is not a link, the plugin will search for the details page link in its child elements or adjacent elements. For relative path links, the plugin will automatically complete the URL to an absolute URL based on the current page URL.

[0012] Preferably, the S5 access strategy is as follows: Sequential access: Open each detail page in the order of the list to avoid putting too much pressure on the target website; Concurrency control: Set reasonable concurrency levels and access intervals, and comply with the website's robots.txt protocol and access frequency limits; Retry on failure: Retry accessing a page multiple times and record the reason for the failure; Dynamic rendering support: For detail pages that require JavaScript rendering, a headless browser or browser extension is used to obtain the fully rendered page content.

[0013] Preferably, the specific content of S6 is as follows: S61. Obtain the complete HTML content of each expert's details page, including the original HTML source code, the rendered DOM tree structure, the text content, table content, list content, and metadata information on the page; S62. Remove script, style, and noscript tags and their contents from the obtained HTML content; remove comments, hidden elements, and ad containers. S63. Extract the main text area, filter the navigation bar, footer, sidebar, and comment section, convert the HTML to structured plain text or Markdown format, and retain headings, paragraphs, and tables.

[0014] Preferably, the prompts in S7 include role definition, task description, field schema definition, contextual semantic rules, and output format constraints; The mapping process in S8 includes field name mapping, data type conversion, value range normalization, information completion, and multi-source fusion.

[0015] Preferably, in S9, the quality verification of the structured expert data includes verification of required fields, format verification, logical verification, duplicate detection, and confidence assessment. For conflicting data, the processing strategies include prioritizing data from more authoritative sources, prioritizing data that has been updated more recently, marking conflicts, and submitting them to human adjudication.

[0016] Preferably, the synchronization methods in S10 include: adding records, updating records, incremental synchronization, and version management; Once synchronization is complete, a data collection report is generated, including a list of collected URLs, the number of experts successfully parsed, the success rate of extraction for each field, statistics on the reasons for failure, and a list of data awaiting manual review.

[0017] Therefore, the present invention employs a semantic understanding-based intelligent collection and structuring method for webpage expert information, which, compared with existing technologies, has the following advantages: 1. This application transforms manual, line-by-line data entry into batch automated data collection, significantly improving efficiency; it eliminates the need to write separate crawler rules for each target website, requiring only one click on the sample element; based on semantic understanding rather than a fixed selector, it can adapt to structural changes of different websites; 2. It can distinguish between the main text content and irrelevant information such as advertisements, navigation, and comments. Through LLM semantic parsing and post-processing verification, it outputs structured and standardized expert data, which can be quickly extended to expert information websites in different fields. Business personnel only need to complete the initial selection and final review, which greatly reduces repetitive work.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 This is an overall flowchart of the present invention, which is a method for intelligent collection and structuring of web page expert information based on semantic understanding. Figure 2 This is a comparison chart showing the time consumption of a semantic understanding-based intelligent collection and structuring method for web page expert information according to the present invention. Figure 3 This is a data flow topology diagram of a semantic understanding-based intelligent collection and structuring method for webpage expert information. Detailed Implementation

[0020] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0021] Example like Figures 1-3 As shown, the present invention provides a method for intelligent collection and structuring of webpage expert information based on semantic understanding, comprising the following steps: S1. The browser plugin is installed in the browser of the business personnel. When the business personnel visit a website containing expert information, the plugin obtains the access permission of the current page through the browser extension API and listens for the page load event. For websites using JavaScript dynamic rendering in S1, the plugin waits for the main page elements to load completely and simulates scrolling, pagination clicks, and other operations when necessary to ensure that the expert list content on the page is fully displayed. It supports the acquisition of complete content from JavaScript dynamically rendered pages through browser plugins and headless browser technology. S2. On the expert list page, business personnel can manually select any list item representing an expert through the interactive interface provided by the browser plugin. In S2, the plugin records information about the user-selected element, including the DOM path, CSS selector, element tag, class name, ID, the element's relative position, size, and visual style characteristics on the page, the text content inside the element and its semantic characteristics, and the structural characteristics of adjacent sibling elements and parent container elements. DOM path: The complete DOM path from the HTML root node to the element; CSS selectors: CSS selectors that are as concise as possible and can locate similar elements; S3 plugins automatically match all expert elements with similar structures and functions in the page DOM tree based on the sample elements selected by the S2 user. Users only need to select one expert element to automatically match all similar elements on the page based on DOM structure, visual features and text semantics, which greatly reduces the threshold for crawler configuration. The specific details of S3 are as follows: S31. Extract the DOM tree structure features of the sample elements, such as tag combination, level depth, number of child nodes, class name pattern, etc., search for other elements with similar DOM structure in the page, and calculate the structural similarity score between the candidate elements and the sample elements. S32. Based on the class, data-* attributes, role attributes, etc. of sample elements, generate reusable CSS selectors, match the set of elements on the page that match the same or similar selectors, and calculate the CSS attribute matching score. S33. Extract the visual features of the sample elements, including element size, border style, background color, font size, layout position, etc., group elements with similar visual features into one category, and calculate the visual similarity score. S34. Perform semantic analysis on the text in the sample elements and their sub-elements to identify whether they include semantic features such as expert names, titles, and research directions. Filter out semantically inconsistent elements such as advertisements, navigation, and recommendation slots, and calculate the semantic similarity score. S35. Based on the scores of multiple dimensions including structural similarity, CSS attribute matching, visual similarity, and semantic matching in S31-S34, the candidate elements are sorted, and the elements with confidence scores higher than the threshold are selected as expert list items. Through the above multi-dimensional matching, the plugin can automatically identify all expert elements on the page without requiring users to manually write complex selectors; by comprehensively utilizing multi-dimensional information such as DOM structure, CSS properties, visual features, and text semantics, the accuracy and robustness of expert element matching are improved. S4. For each expert element identified in S3, the plugin automatically extracts the hyperlinks (href attribute), deduplicates and filters all extracted expert detail page links (e.g., excluding external advertising links, duplicate links, and non-detail page links), and generates a list of detail page URLs to be accessed; In S4, if the expert element itself is not a link, the plugin will search for the details page link in its child elements or adjacent elements. For relative path links, the plugin will automatically complete the URL to an absolute URL based on the current page URL. S5. The plugin sends the list of detail page URLs to the cloud server or allows the plugin to open each detail page in batches in the browser according to the access policy. The access policy for S5 is as follows: Sequential access: Open each detail page in the order of the list to avoid putting too much pressure on the target website; Concurrency control: Set reasonable concurrency levels and access intervals, and comply with the website's robots.txt protocol and access frequency limits; Retry on failure: Retry accessing a page multiple times and record the reason for the failure; Dynamic rendering support: For detail pages that require JavaScript rendering, a headless browser or browser extension is used to obtain the fully rendered page content; S6. Obtain the complete HTML content of each expert's details page and preprocess the obtained HTML; The specific details of S6 are as follows: S61. Obtain the complete HTML content of each expert's details page, including the original HTML source code, the rendered DOM tree structure, the text content, table content, list content, and metadata information such as page title, meta tags, and OpenGraph tags. S62. Remove script, style, noscript tags and their contents from the obtained HTML content, and remove comments, hidden elements, and ad containers; S63. Extract the main text area, filter out noise from the navigation bar, footer, sidebar, comment section, etc., convert the HTML into structured plain text or Markdown format, and retain hierarchical information such as headings, paragraphs, and tables; S7. Send the preprocessed webpage text content to the cloud-based large language model, guide the model to identify and extract expert information fields through structured prompts; use LLM to understand the semantics of webpage content and intelligently distinguish expert text information from irrelevant noise such as advertisements, navigation, and comments; The prompts in S7 include role definition, task description, field schema definition, contextual semantic rules, and output format constraints; Role Definition: Clearly define LLM as an "expert information extraction expert"; Task Description: Extract relevant information about experts from web page text; Field Schema Definition: Defines the fields to be extracted, for example: Basic information: Name, gender, date of birth, professional title, position, education, degree; Work unit: Company name, department, type of unit; Research direction: research field, research interests, professional expertise; Academic achievements: papers, patents, books, research projects, awards; Contact information: office phone number, email address, office address (determined whether to extract funds based on compliance requirements); Talent titles: Academician, Outstanding Young Scientist, Yangtze River Scholar, Thousand Talents Program, etc.; Social positions: positions in academic organizations, journal editorial board members, reviewer experience, etc.; Contextual semantic rules: Distinguish between the expert's personal information and information of other team members; Distinguish between current employer and previous employer; Distinguish between the first author, corresponding author, and participants in academic research; Identify and filter irrelevant information such as advertisements, navigation, recommendations, and comments on the page.

[0022] Output format constraints: The LLM must be output in JSON object format. Missing fields should be represented by null values ​​or placeholders, and multi-value fields should be represented by arrays. S8. Receive JSON data returned by LLM and perform field mapping and format standardization according to the company's internal expert data model; The mapping process in S8 includes field name mapping, data type conversion, value range normalization, information completion, and multi-source fusion. Field name mapping: Maps LLM output field names to standard field names in the expert database; Data type conversion: such as standardizing date formats, cleaning numeric fields, and splitting array fields; Value range standardization: such as mapping professional titles to the standard professional title system, aligning unit names with the standard institution database, and aligning research directions with the discipline classification system; Information completion: For missing fields, information can be inferred based on existing information or marked as needing to be completed; Multi-source fusion: When the same expert has multiple source pages, information from different sources is merged, duplicates are removed, and the most complete and up-to-date data is retained; S9. Perform quality verification on the expert data after S8 structuring, and provide processing strategies for conflicting data; S9 performs quality checks on structured expert data, including required field checks, format checks, logical checks, duplicate detection, and confidence level assessment. Required field validation: Check whether key fields such as name and organization are missing; Format validation: Check if the formats of fields such as email, phone number, and date are correct; Logical checks: Educational experience should precede work experience; multiple appointments for the same expert should not conflict; professional title should match age and years of service; Duplicate detection: Compare the similarity of name, affiliation, and research direction with the existing expert database; identify potentially duplicate expert records and prompt for merging or manual confirmation; Confidence assessment: Calculate data confidence based on LLM output confidence, field completeness, and validation results; low-confidence data proceeds to manual review. For conflicting data, the processing strategies include prioritizing data from more authoritative sources, prioritizing data that has been updated more recently, marking conflicts, and submitting them to human adjudication. S10. Synchronously write the verified expert data into the company's expert database and generate a collection report; The synchronization methods in S10 include: adding records, updating records, incremental synchronization, and version management. Add a new record: For experts that do not exist in the database, insert a new record; Update log: For existing experts, update fields based on data freshness and source credibility; Incremental synchronization: Only newly added or changed data is synchronized, improving synchronization efficiency; Version management: Retain historical versions of expert information to facilitate tracing the change process; Once synchronization is complete, a data collection report is generated, including a list of collected URLs, the number of experts successfully parsed, the success rate of extraction for each field, statistics on the reasons for failure, and a list of data awaiting manual review.

[0023] Typical application scenarios are as follows: Information collection of university experts: Information on professors, associate professors, and researchers is collected from the faculty pages of various universities; Building a talent pool for research institutions: collecting information on researchers from the official websites of research institutes and the pages of key laboratories; Expanding the expert database of academic societies and associations: collecting information on expert committee members from the official websites of various academic societies and associations; Talent Program Selectees Collection: Information on selected experts is collected from the announcement pages of various talent programs. The expert review database is dynamically updated: the target website is regularly revisited to update information on changes in experts' professional titles, research directions, academic achievements, etc.

[0024] In the specific implementation process, a review service company needed to expand its expert database and collect expert information from the Computer Science Department of University A's official website's faculty page. This faculty page is a dynamically rendered JavaScript webpage, with the expert list displayed in paginated format. Clicking on each expert entry redirects to an independent details page, which includes basic expert information, research interests, academic achievements, etc. The page also contains noisy content such as navigation bars, side announcements, and advertisements. The present invention's semantic understanding-based intelligent collection and structuring method for webpage expert information was used to carry out the collection work. The complete execution flow is as follows: Plugin Deployment and Page Monitoring: Business personnel install the browser scraping plugin that accompanies this solution in the Chrome browser and access the faculty list webpage of the Computer Science Department of University A. The plugin calls the browser extension API to obtain page permissions and listens for page load events. This faculty page is dynamically rendered using JavaScript. Initially, only the first 10 experts are loaded; the remaining content requires scrolling and clicking the pagination button to load. The plugin waits for the basic DOM rendering of the page to complete, automatically simulates scrolling down the page, triggers lazy loading, and then simulates clicking the pagination button, looping through and loading all four pages of the expert list to ensure that all expert entries are rendered and displayed completely.

[0025] User selects a sample expert element: On the fully loaded expert list page, the business user opens the plugin's interactive interface and selects the expert list item "Professor Zhang XX" as a sample element. The plugin automatically records all features of this sample element. DOM path, CSS selector, tag, class name, element ID; Visual styles such as element size, relative position on the page, font, and background color; Internal text: “Zhang XX, Professor, Doctoral Supervisor”, extract semantic features (including name and title); The DOM structure of the parent container and adjacent sibling nodes.

[0026] The plugin automatically matches all expert list items from multiple dimensions, executing a multi-dimensional scoring and matching algorithm to identify all expert entries on the page. Compare the DOM hierarchy, number of child nodes, and tag combinations of all candidate elements on the page to calculate the DOM structure similarity score; Candidate CSS selectors are generated based on the class and role attributes of the samples, candidate elements that match the style attributes are filtered, and CSS matching scores are output. Compare the size, layout, and font style of candidate elements to filter out modules with significant visual differences, such as side ads and bulletin boards on the page. Identify whether candidate element text contains expert-related semantics such as "professor / associate professor / researcher" and filter out semantically mismatched entries such as news headlines and enrollment notices; The four scores were weighted to calculate the overall confidence level. A threshold of 0.7 was set, and items with a confidence level ≥ 0.7 were retained. Finally, 42 expert list items were identified on the page, and 6 interfering elements, including page advertisements and notifications, were removed.

[0027] Extracting and processing the details page URLs: The plugin iterates through the 42 identified expert list elements. For some expert entries that are not inherently link tags, the plugin traverses down the child elements to find the hyperlinks corresponding to the expert names. Some links are relative paths; the plugin automatically completes these using the current page domain name, converting them to absolute URLs. Deduplication filtering is performed to remove invalid redirects, ultimately generating a list of URLs for all 42 expert details pages to be accessed.

[0028] Batch retrieve detail pages according to access policies, submit the URL list to the cloud server, and configure the access policies: Concurrency is set to 2, access interval is 1.5 seconds, access is sequential, and the website robots protocol is followed. If a page fails to load, attempt to retry 3 times and record the reason for the failure. The expert details page for University A is a dynamically rendered page using JavaScript, which utilizes a headless browser to render the page and retrieve the rendered DOM. Of the 42 links accessed, two pages experienced server timeouts but were successfully retrieved after retries; the remaining 40 pages were accessed successfully on the first attempt.

[0029] Perform preprocessing on the complete rendered HTML returned for each details page: Obtain the original HTML source code, render the DOM, and extract page text, tables, and metadata. Remove script, style, and noscript tags; clear webpage comments, pop-up ad containers, and hidden modules from the page. Automatically identify the main text area, filter the top navigation bar, bottom footer, side enrollment announcement, and comment section, convert the main text to Markdown format, and retain the expert's educational experience table and project list; after noise filtering, output a clean expert text for delivery to the large model.

[0030] Input the preprocessed Markdown text into the cloud-based large model, and input structured suggestions: [Role] You are an expert information extraction expert; [Task] Extract expert information from webpage text;

Schema Fields

[0031] The large model parses web page content based on prompt words and outputs JSON structured expert data. For example, it outputs fragments for Professor Zhang XX's page, as shown in Table 1.

[0032] Table 1 shows the output snippets for Professor Zhang XX's page;

[0033] Receive the JSON returned by the LLM, connect it to the enterprise's internal expert database standard model, and perform processing: Field mapping: Map the model output "Department" to the internal library standard field "Affiliated Department"; Data type conversion: Convert papers and research directions to array format; Value range normalization: the professional titles "professor, doctoral supervisor" are decomposed and mapped to the internal standard dictionary; the institution "University A" is aligned with the full name of the institution's standard library; Multi-source fusion: Some experts have information on multiple pages, so the information is merged and conflicting old data is removed.

[0034] Multi-dimensional validation was performed on 42 pieces of structured expert data: Required field validation: Check that the name and work unit are not empty; Format validation: Verify the validity of the email address format; Logical verification: Educational experience predates work experience; Duplicate detection: Compared with the company's existing expert database, two experts were identified as already existing in the database; Confidence assessment: Key information is missing in 3 data points, resulting in low confidence. Example of conflict handling: One expert's title is listed as "Associate Professor" on webpage A, while another public page displays "Professor". The system determines that the latter source is more up-to-date, prioritizes the updated version, marks it as conflicting, pushes it to the manual review queue, and marks three low-confidence incomplete data entries as awaiting manual review.

[0035] Data is imported into the database and a collection report is generated: Data synchronization: 37 new expert records that passed verification were added to the database; 2 existing expert records were updated, version management was enabled, and historical information was retained; 3 data records pending review were not automatically added to the database.

[0036] The output acquisition report is as follows: Total number of URLs collected: 42; Successfully parsed: 39 records; Automatically entered into database: 37 records; Pending manual review: 3 records; Field extraction success rates: Name 100%, Research direction 92.8%, Email 78.5%; Failure statistics: 0 links failed permanently; The attached output includes a list of experts awaiting manual review and a complete list of original URLs.

[0037] The business personnel reviewed the report, manually confirmed and corrected the three pieces of expert data pending review, and completed the data collection and entry into the database by experts from the Computer Science Department of University A.

[0038] Three months later, staff reused the plugin and accessed the faculty page again. The system compared the latest webpage content with the database version, identified two experts who had been promoted and five new teachers who had joined the faculty, performed incremental synchronization, updated the expert database, and completed the dynamic maintenance of expert information.

[0039] Traditional manual collection of information on these 42 experts would take 4-5 hours, while this solution takes only about 12 minutes from sample selection to report generation, significantly reducing manpower. It eliminates the need to write XPath and crawler rules for University A's website; after a website redesign, only one sample selection is required for adaptation. Through multi-dimensional matching and LLM semantics, it automatically filters advertisements and navigation noise, outputting structured data that conforms to the internal standard library. A complete quality verification, conflict marking, and manual review mechanism ensures the quality of the expert data in the database.

[0040] Therefore, the present invention adopts a semantic understanding-based intelligent collection and structuring method for web page expert information, which effectively solves the problems of low efficiency of traditional manual input, high maintenance cost of traditional crawlers, large differences in web page structure, and a lot of noise interference. It provides an efficient and reliable technical means for review service companies to quickly expand and update expert databases.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for intelligent collection and structuring of webpage expert information based on semantic understanding, characterized in that, Includes the following steps: S1. The browser plugin is installed in the browser of the business personnel. When the business personnel visit a website containing expert information, the plugin obtains the access permission of the current page through the browser extension API and listens for the page load event. S2. On the expert list page, business personnel can manually select any list item representing an expert through the interactive interface provided by the browser plugin. S3, the plugin, is based on the sample elements selected by the S2 user and automatically matches all expert elements with similar structures and functions in the page DOM tree; S4. For each expert element identified in S3, the plugin automatically extracts the hyperlinks, deduplicates and filters all extracted expert detail page links, and generates a list of detail page URLs to be accessed. S5. The plugin sends the list of detail page URLs to the cloud server or allows the plugin to open each detail page in batches in the browser according to the access policy. S6. Obtain the complete HTML content of each expert's details page and preprocess the obtained HTML; S7. Send the preprocessed web page text content to the large language model in the cloud, and guide the model to recognize and extract expert information fields through structured prompt words; S8. Receive JSON data returned by LLM and perform field mapping and format standardization according to the company's internal expert data model; S9. Perform quality verification on the expert data after S8 structuring, and provide processing strategies for conflicting data; S10. Synchronously write the verified expert data into the company's expert database and generate a collection report.

2. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 1, characterized in that: In S1, for websites that use JavaScript dynamic rendering, the plugin will wait for the main elements of the page to finish loading to ensure that the content of the expert list on the page is fully displayed.

3. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 2, characterized in that: The specific content of S2 is as follows: The plugin records information about the element selected by the user, including the DOM path, CSS selector, element tag, class name, ID, the element's relative position, size, and visual style characteristics on the page, the text content inside the element and its semantic characteristics, and the structural characteristics of adjacent sibling elements and parent container elements.

4. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 3, characterized in that: The specific details of S3 are as follows: S31. Extract the DOM tree structure features of the sample elements, search for other elements with similar DOM structures in the page, and calculate the structural similarity score between the candidate elements and the sample elements. S32. Based on the class, data-*, and role attributes of the sample elements, generate reusable CSS selectors, match the set of elements on the page that match the same or similar selectors, and calculate the CSS attribute matching score. S33. Extract the visual features of the sample elements, group elements with similar visual features into one category, and calculate the visual similarity score. S34. Perform semantic analysis on the text in the sample elements and their sub-elements, identify whether they contain semantic features, filter out elements that do not match the semantics, and calculate the semantic similarity score. S35. Based on the scores from multiple dimensions including structural similarity, CSS attribute matching, visual similarity, and semantic matching from S31 to S34, the candidate elements are ranked, and elements with a confidence level higher than the threshold are selected as expert list items.

5. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 4, characterized in that: In S4, if the expert element itself is not a link, the plugin will search for the details page link in its child elements or adjacent elements. For relative path links, the plugin will automatically complete the URL to an absolute URL based on the current page URL.

6. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 5, characterized in that: The access policy for S5 is as follows: Sequential access: Open each detail page in the order of the list to avoid putting too much pressure on the target website; Concurrency control: Set reasonable concurrency levels and access intervals, and comply with the website's robots.txt protocol and access frequency limits; Retry on failure: Retry accessing a page multiple times and record the reason for the failure; Dynamic rendering support: For detail pages that require JavaScript rendering, a headless browser or browser extension is used to obtain the fully rendered page content.

7. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 6, characterized in that: The specific details of S6 are as follows: S61. Obtain the complete HTML content of each expert's details page, including the original HTML source code, the rendered DOM tree structure, the text content, table content, list content, and metadata information on the page; S62. Remove script, style, and noscript tags and their contents from the obtained HTML content; remove comments, hidden elements, and ad containers. S63. Extract the main text area, filter the navigation bar, footer, sidebar, and comment section, convert the HTML to structured plain text or Markdown format, and retain headings, paragraphs, and tables.

8. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 7, characterized in that: The prompts in S7 include role definition, task description, field schema definition, contextual semantic rules, and output format constraints; The mapping process in S8 includes field name mapping, data type conversion, value range normalization, information completion, and multi-source fusion.

9. The method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 8, characterized in that: S9 performs quality checks on structured expert data, including required field checks, format checks, logical checks, duplicate detection, and confidence level assessment. For conflicting data, the processing strategies include prioritizing data from more authoritative sources, prioritizing data that has been updated more recently, marking conflicts, and submitting them to human adjudication.

10. A method for intelligent collection and structuring of webpage expert information based on semantic understanding according to claim 9, characterized in that: The synchronization methods in S10 include: adding records, updating records, incremental synchronization, and version management. Once synchronization is complete, a data collection report is generated, including a list of collected URLs, the number of experts successfully parsed, the success rate of extraction for each field, statistics on the reasons for failure, and a list of data awaiting manual review.

Images