Content extraction and AIGC fusion generation method and system

Abstract

The invention discloses a content extraction and AIGC fusion generation method, which comprises the following steps: in response to a query request input by a user, capturing webpage content by using a crawler technology and generating a content template by using AIGC; according to the semantic similarity between the semantic vector of the user intention corresponding to the query request and the semantic vector of the webpage content, screening out effective webpage content related to the user intention; identifying a keyword code in the AIGC generation content template as a target content semantic vector, and screening out the webpage content matched with the keyword according to the semantic similarity between the semantic vector of the effective webpage content and the target content semantic vector; the webpage content matched with the keyword is fused with the content template, and generated content corresponding to the query request is obtained; according to the method, the draft template is generated by the AIGC according to the intention of the user, then the most matched content is screened from the webpage content to be used for supplementing and correcting the draft template under the guidance of the template, and finally the fusion output of the generated content and the real information is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for web page content extraction and AIGC fusion generation. Background Art

[0002] In the power industry, with the rapid growth of smart grids, equipment digitization, and operations and maintenance data, power engineers, operators, and designers are increasingly demanding real-time, high-quality information. However, currently, most power equipment data, design materials, product manuals, and policy standards are scattered across multiple manufacturer websites, industry forums, technical portals, and policy databases. This fragmented access and inefficient querying significantly impact engineers' data retrieval efficiency and the quality of their analytical decisions.

[0003] Current mainstream content acquisition methods usually rely on manual search or traditional crawler technology to capture web page information. However, traditional crawlers face bottlenecks in dynamism, accuracy, and semantic understanding in the power industry. The reason is that traditional crawlers are mostly based on static HTML structure rules. When faced with JS dynamic rendering pages commonly seen in the power industry (such as product configuration parameter comparison pages and interactive drawing browsers), it is difficult to extract valid data. In addition, traditional crawlers lack the ability to understand contextual semantics during the content screening process, resulting in redundant or noisy content being captured. The semantics of professional terms, multi-field combination expressions, and other expressions in power technical documents are difficult to accurately understand and extract.

[0004] Combining AIGC (Artificial Intelligence Generated Content) technology can address these shortcomings through data reconstruction and semantic understanding. Currently, existing public literature proposes solutions that integrate web page extraction with AIGC. For example, patent application publication number CN117350783A proposes integrating crawler technology and AIGC to automatically generate advertising schemes, and patent application publication number CN117421343A proposes combining a crawler framework with AIGC to recommend cultural and tourist attractions. However, these fusion solutions all use a hybrid framework in which crawlers are responsible for collecting raw data, followed by the AIGC module for cleaning, enhancement, and generation. These hybrid frameworks lack the ability to structure content related to user question intent and are unable to achieve reverse-guided content extraction, which aims to "find useful answers from questions." These solutions often suffer from problems such as loose structure, unclear logic, or distorted facts. Distorted content generated by the power question-and-answer platform can trigger systemic risks and even lead to safety incidents and significant economic losses.

[0005] In the paper "Design and Implementation of a Program for Obtaining Multi-source AIGC Detection Results Based on Web Crawler-Assisted Editing and Publishing, Fan Cuili, Journal Editing Series", it is proposed to use web crawlers to assist in detecting the contribution rate of AIGC in generated content. It focuses on the detection of AIGC contribution rate rather than the generation of AIGC content.

[0006] Therefore, there is an urgent need for a solution that effectively combines web crawling with AIGC to achieve high-quality power knowledge generation and question-answering content construction. Summary of the Invention

[0007] The technical problem to be solved by the present invention is how to improve the accuracy of content generated by artificial intelligence.

[0008] The present invention solves the above technical problems through the following technical means:

[0009] A method for content extraction and AIGC fusion generation is proposed, the method comprising:

[0010] Respond to user input queries, use crawler technology to crawl web content, and use AIGC to generate content templates;

[0011] Filter out valid web page content related to the user's intent based on the semantic similarity between the user's intent semantic vector corresponding to the query request and the semantic vector of the web page content;

[0012] Identify keywords in the AIGC-generated content template and encode them into target content semantic vectors. Filter out web page content that matches the keywords based on the semantic similarity between the semantic vectors of valid web page content and the semantic vectors of target content.

[0013] The web page content that matches the keyword is integrated with the content template to obtain the generated content corresponding to the query request.

[0014] Furthermore, the crawling of web page content using crawler technology includes:

[0015] When a web page is first visited, a web page structure template is created based on the DOM tree;

[0016] When accessing the web page subsequently, the structure of the currently accessed page is compared with the web page structure template, and the crawling strategy is updated when it is determined that the web page structure difference exceeds the set threshold.

[0017] Furthermore, the crawling strategy update process includes:

[0018] Update matching rules and field paths, and retrain the field recognition model;

[0019] Combined with user feedback indicators, dynamically adjust crawling frequency, field priority, and content area positioning strategies.

[0020] Furthermore, the method of screening out valid web page content related to the user intent based on the semantic similarity between the user intent semantic vector corresponding to the query request and the semantic vector of the web page content includes:

[0021] Use the pre-trained BERT model to semantically encode the query request entered by the user and generate the user intent semantic vector;

[0022] Perform semantic encoding on the captured multimodal web page content using the BERT model to generate a semantic vector for the web page content;

[0023] The cosine similarity between the user intention semantic vector and the semantic vector of the web page content is calculated, and the valid web page content related to the user intention is filtered out based on the similarity score.

[0024] Furthermore, after filtering out valid webpage content related to the user's intention, the method further includes:

[0025] The context window of the filtered valid web page content is expanded, and the web page content structure is reconstructed and returned.

[0026] Furthermore, the step of identifying keywords in the AIGC-generated content template and encoding them into target content semantic vectors includes:

[0027] Use the NER model to extract key semantic units in the content template;

[0028] Combining part-of-speech analysis with the BERT-based attention weighting mechanism, we can identify keywords corresponding to key semantic units from content templates.

[0029] The keywords are encoded into target content semantic vectors.

[0030] Furthermore, the method of screening out web page content that matches the keyword based on the semantic similarity between the semantic vector of the effective web page content and the semantic vector of the target content includes:

[0031] The cosine similarity between the semantic vector of the effective web page content and the semantic vector of the target content is calculated, and the web page content matching the keyword is screened out based on the similarity score.

[0032] Furthermore, the step of fusing the webpage content that matches the keyword with the content template to obtain generated content corresponding to the query request includes:

[0033] Merging webpage content matching the keyword with the content template according to pre-configured fusion rules, wherein the fusion rules include field priority, overlay conditions, and insertion position markers;

[0034] If the webpage content contains more accurate information than the content template, the information is overwritten into the corresponding fragment information in the content template;

[0035] If the web page content includes key information fields that are not included in the content template, the key information fields are inserted into the corresponding positions of the content template.

[0036] Furthermore, after fusing the webpage content matching the keyword with the content template to obtain generated content corresponding to the query request, the method further includes:

[0037] The BART model is called to perform language polishing on the generated content to obtain the final generated content.

[0038] In addition, the present invention also proposes a content extraction and AIGC fusion generation system, which includes:

[0039] The first response module is used to respond to the query request input by the user and crawl the web page content using crawler technology;

[0040] The second response module is used to respond to the query request input by the user and generate a content template using AIGC;

[0041] A primary screening module is used to screen out valid web page content related to the user's intent based on the semantic similarity between the user's intent semantic vector corresponding to the query request and the semantic vector of the web page content;

[0042] The secondary screening module is used to identify keywords in the content template generated by AIGC and encode them into target content semantic vectors. Based on the semantic similarity between the semantic vectors of valid web page content and the semantic vectors of target content, the module selects web page content that matches the keywords.

[0043] The fusion module fuses the web page content that matches the keyword with the content template to obtain the generated content corresponding to the query request.

[0044] The advantages of the present invention are:

[0045] (1) The present invention preliminarily screens out valid web page content related to user intent through the similarity between the semantic vector of user intent and the semantic vector of web page content, which can effectively exclude a large amount of web page content irrelevant to user intent, and construct a candidate set with semantic proximity to the user query request, ensuring that the subsequent AIGC reverse guidance process is carried out within the scope of semantically credible materials, improving the overall efficiency and generation accuracy of the system, and reducing the processing load of the subsequent reverse guidance process; and by realizing secondary screening and fact fusion of web page content based on the semantic similarity between the semantic vector of valid web page content and the semantic vector of target content, thereby ensuring the authenticity and integrity of the final output content; the two similarity calculations complement each other and are respectively used in the "coarse screening" and "fine screening" stages of the content extraction process. A closed-loop mechanism of "generation → guidance → integration" is constructed. AIGC generates a draft content template based on user intention, crawls web page content based on user intention, and then uses the content template as a guide to screen the most matching content in the web page content to supplement and correct the content template, ultimately achieving the integrated output of generated content and real information. This ensures that the generated content is both semantically consistent and supported by real data, avoiding the problems of fiction and inconsistency in traditional AIGC, and improving generation quality, factual credibility and traceability.

[0046] (2) The present invention performs preliminary screening of web page content based on a semantic vector alignment mechanism, breaking away from the limitations of traditional keywords and HTML structure rules. It uses deep semantic models such as BERT to achieve high-dimensional matching between user intent vectors and web page content fragments, and combines the context window expansion strategy to achieve paragraph-level relevance enhancement and completeness completion, effectively improving the accuracy and consistency of content extraction.

[0047] (3) The web page structure change detection mechanism and user interaction feedback mechanism are integrated to automatically identify changes in web page structure and adjust crawler rules synchronously. Combined with the content scoring model driven by user behavior (such as click-through rate and bounce rate), the extraction priority is dynamically optimized to achieve real-time iterative updates of content extraction and personalized generation.

[0048] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 1 is a flow chart of a method for content extraction and AIGC fusion generation proposed in one embodiment of the present invention;

[0050] Figure 2 It is a structural diagram of a content extraction and AIGC fusion generation system proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] like Figure 1 As shown, the first embodiment of the present invention proposes a method for content extraction and AIGC fusion generation, which includes the following steps:

[0053] S10, responding to a query request input by a user, crawling web page content using crawler technology and generating a content template using AIGC;

[0054] S20, screening out valid web page content related to the user intent based on the semantic similarity between the user intent semantic vector corresponding to the query request and the semantic vector of the web page content;

[0055] S30, identifying keywords in the AIGC-generated content template and encoding them into target content semantic vectors, and screening out web page content that matches the keywords based on the semantic similarity between the semantic vectors of valid web page content and the target content semantic vectors;

[0056] S40: Merge the webpage content that matches the keyword with the content template to obtain generated content corresponding to the query request.

[0057] It should be noted that AIGC (AI Generated Content) technology uses a pre-trained large language model (such as GPT-3) to perform semantic understanding of user questions. Based on user needs or interest preferences, this embodiment first generates a preliminary content structure draft by the AIGC model (such as GPT-3, DALL-E), which is the content template described in this embodiment. Then, it compares and integrates the content with the actual paragraphs extracted from the web page, realizing a closed loop of "generation-extraction-fusion", thereby generating personalized text, image or video content with reasonable structure, unified semantics and traceability. In addition, the web page content extraction link is screened based on the user's semantic intent, retaining only content that is highly relevant to the generation target, ensuring that the generated content meets user needs; the generated content no longer relies entirely on the free imagination of the model, but matches the real web page information through semantic alignment, effectively avoiding the "hallucination"-style fabrication problem of AIGC; the system integrates the AIGC generation results with the real web page materials in the output stage, and the generated content has both the advantages of natural language expression and clear and traceable factual support.

[0058] The traditional content generation process usually adopts the method of "first crawling web page content, then performing AI text generation", which often has problems such as loose structure of generated content, unclear logic or distorted facts. The present invention proposes the idea of ​​"pre-generation". Although the web page content is pre-crawled in the system process, the present invention does not directly rely on the crawled content to complete the output during the content generation stage. Instead, it first generates an AIGC draft, extracts its structure and semantic elements, and then uses the generated modality as a guide to reverse match and filter web page content with consistent semantics for filling and verification. Therefore, whether the crawled web page content is used depends on the semantic similarity evaluation between the generated modality and the web page content, rather than the decisive influence of the web page itself on the generated content. This mechanism embodies the strategy of controlling information extraction based on "generation intention". Different from the traditional content construction logic of first crawling and then generating, it can not only capture key data, but also ensure that the crawled content is highly relevant to user needs, thereby significantly improving the crawling accuracy and content quality, and better meeting the actual needs of the power industry.

[0059] In addition, through the coordination of "coarse screening" and "fine screening", the processing load is reduced, which not only improves the efficiency of data processing, but also optimizes the quality of content and user satisfaction. Therefore, relevant content can be extracted and generated in real time according to the user's specific instructions and preferences, greatly improving the relevance and personalization of the content and the user's interactive experience.

[0060] It should be noted that the content extraction and AIGC fusion generation method proposed in this embodiment is suitable for scenarios with high requirements for the authenticity of AI-generated content. For example, it can be used in the power industry to generate personalized power question-and-answer content. Taking the intelligent recommendation scenario of power grid equipment as an example, the user inputs "Recommend several anti-corrosion transmission towers suitable for use in coastal high-humidity environments." The system generates a content draft through AIGC, for example: "Transmission towers suitable for coastal areas generally use salt spray corrosion-resistant materials, such as hot-dip galvanized steel towers or composite towers. Among them, single-circuit straight tower structures are commonly used for transmission lines of 220kV and below, which have strong corrosion resistance and wind load adaptability."

[0061] The system extracts core semantic information from this content, such as keywords like "coastal area," "salt spray corrosion resistance," "hot-dip galvanized steel tower," "220kV," and "single-circuit line," and encodes this information into a target content semantic vector, V_target. It then searches a web database for web content from power equipment manufacturers' official websites or industry specification documents, such as "A certain model HDG-Z220 transmission tower has passed the GB / T 36284 salt spray corrosion test standard and is suitable for use in typhoon-prone coastal areas." The system then uses semantic similarity to determine the degree of match between the web content semantic vector and V_target.

[0062] If the system finds a close match between webpage content and material descriptions, usage scenarios, voltage levels, and other aspects, it uses that content to complement, replace, or correct AIGC-generated content. The resulting recommended text not only retains the AIGC-generated style, but also ensures the authenticity of the source and accuracy of the information, which can be traced back to specific webpage paragraphs.

[0063] Furthermore, this embodiment uses customized web crawler technology to crawl target web page content from the Internet, download the web page by accessing the specified URL, and then extract useful information such as text, images, video links, etc., to ensure that sufficient and relevant raw data can be obtained for further processing.

[0064] Specifically, the specific crawling implementation steps of the customized web crawler are as follows:

[0065] (1) Goal setting and crawling initialization

[0066] Custom crawlers first set crawling targets according to user needs, including:

[0067] Specify the target website or data source: Users can specify the web page URL or specific website (such as e-commerce website, news website, etc.) to be crawled according to their needs.

[0068] Set crawling rules: Users can set crawling rules by customizing parameters, such as crawling frequency, link mode of specific pages (such as paged crawling), etc.

[0069] Crawling target content type: Define the target content type to be crawled, such as text, image, video, etc., to ensure that the crawler only crawls and stores relevant information.

[0070] It should be noted that customized crawlers can specifically select the web pages and data fields to be crawled based on these settings.

[0071] (2) Page loading and content crawling

[0072] Static web page crawling: For static HTML pages, the crawler directly requests the target URL and obtains the page content. The crawler usually uses an HTTP request library (such as requests) to request the page source code (HTML) and parse it.

[0073] Dynamic web crawling: Custom crawlers use tools (such as Selenium) to simulate browser behavior, automatically execute JavaScript on the web page, let the page render completely, and then extract the page content. This ensures that the crawler not only crawls static content but also dynamically loaded data.

[0074] Simulating user interactions: Customized crawlers can also simulate user actions (such as clicks, scrolling, and logging in) to capture dynamic data triggered by user interactions. For example, a custom crawler can simulate a "Load More" button or pagination button on an e-commerce website to load and extract more data.

[0075] It should be noted that the core task of a custom crawler is to crawl web page content. For dynamic pages that include JavaScript rendering, traditional crawlers cannot crawl content dynamically generated by AJAX or JavaScript. Compared to traditional crawlers, this embodiment supports not only static web pages, but also dynamic web pages (such as content dynamically loaded by JavaScript or AJAX).

[0076] (3) Content extraction and structured data storage

[0077] Custom crawlers not only crawl page content, but also structure the crawled content and perform data cleaning as needed. This part includes:

[0078] Text extraction: Use the HTML parsing library to parse the text information on the page (such as product descriptions, news content, comments, etc.), extract it from the HTML, and convert it into a structured data format (such as JSON, CSV).

[0079] Image extraction: The crawler can extract the image URL in the page and download the related image data, or just save the image metadata (such as the image link, description, etc.).

[0080] Video extraction: The crawler is also able to extract the embed code or URL address of the video and store it as related information.

[0081] Data cleaning: The content extracted by crawlers often contains unnecessary parts (such as advertisements, navigation menus, etc.). Custom crawlers clean the data by removing irrelevant content and formatting information such as dates and prices to ensure data consistency and accuracy.

[0082] Storage and indexing: The extracted data will be stored in a database (such as MongoDB, Elasticsearch, etc.) for subsequent query and analysis.

[0083] It should be noted that this embodiment introduces browser automation technology (such as Selenium) into the traditional crawler technology framework, which can simulate user behavior and execute JavaScript scripts to fully load web page content, including dynamically rendered text, images and videos. In terms of web crawling, browser automation tools are used to simulate user behavior and can crawl dynamically loaded page content. The system supports multiple rounds of interactive simulation and can accurately capture deeply nested information and real-time updated content in complex web pages. This mechanism significantly improves the compatibility with modern web pages and the integrity of data extraction, breaking through the technical bottleneck of traditional crawlers in dynamic web page processing.

[0084] As a further preferred technical solution, the customized crawler of this embodiment can dynamically identify structural changes of web pages through machine learning algorithms or regular expressions, and adjust crawling rules as needed. For example, if the layout of a web page changes, the crawler can intelligently identify the new structure and update the crawling strategy to ensure that no data is missed. In the above step S10, crawling web page content using crawler technology specifically includes:

[0085] When a web page is first visited, a web page structure template is created based on the DOM tree;

[0086] When accessing the web page subsequently, the structure of the currently accessed page is compared with the web page structure template, and the crawling strategy is updated when it is determined that the web page structure difference exceeds the set threshold.

[0087] Specifically, when a web page is visited for the first time, the web page structure modeling is performed, and a web page structure template S0 is established by recording the DOM tree structure, key field XPath path, tag semantic layout, etc.;

[0088] In subsequent visits, structural change detection is performed by comparing the current page structure S t The node tree edit distance (Tree Edit Distance) from the original template S0 is combined with indicators such as field missing rate and node offset ratio to determine whether a structural change has occurred. Specifically:

[0089] 1) Strategy update mechanism: When the structural difference exceeds the set threshold (such as edit distance > 20% and field missing > 30%), the system automatically calls the structure adaptation module to update the matching rules and field paths and retrain the field recognition model (such as the CRF model or XPath pattern generator for table recognition).

[0090] 2) Continuously optimize crawling strategies: Based on feedback indicators such as user click-through rate, crawling accuracy, and crawling failure rate, the system dynamically adjusts crawling frequency, field priority, and content area positioning strategies to achieve "effect-oriented" crawler behavior control.

[0091] It should be noted that, in this embodiment, the customized crawler can adjust the crawling frequency according to the update frequency of the web page or the needs of the user to ensure that the latest web page content is obtained. For example, a news website may require a higher frequency of crawling, while a website with a lower frequency of updating commodity prices can reduce the crawling frequency. The crawler can also adjust the crawling strategy based on user feedback (such as the click-through rate and accuracy of the crawled data). For example, the crawler intelligently identifies which pages have more valuable content based on the content of the pages visited by the user, and gives priority to crawling these pages. After adopting this real-time crawling technology, the crawler can automatically capture all dynamically changing data, including text, pictures, and videos that are updated in real time.

[0092] As a further preferred technical solution, consider that custom crawlers may encounter various problems during the crawling process (such as network interruptions, page loading failures, etc.). To this end, custom crawlers are usually configured with failure retry mechanisms and exception handling strategies to ensure that the crawling task is not interrupted by a single error. Common exception handling strategies include:

[0093] Automatic retry mechanism: When a page fails to be crawled, the crawler will automatically try to re-crawl according to the preset retry strategy until the crawl is successful or the maximum number of retries is reached.

[0094] Logging and alerting mechanisms: The crawler records any anomalies during the crawling process and generates log reports. When an anomaly occurs, the system triggers an alert so that developers can address it promptly.

[0095] As a further preferred technical solution, step S20: screening out valid web page content related to the user intent based on the semantic similarity between the user intent semantic vector corresponding to the query request and the semantic vector of the web page content, specifically includes the following steps:

[0096] S21. Use the pre-trained BERT model to semantically encode the query request input by the user and generate a user intent semantic vector;

[0097] It should be noted that this embodiment's web content extraction mechanism, based on user semantic intent vector matching, differs from traditional extraction methods based on keywords or HTML structure rules. This approach significantly improves the semantic relevance between web content and user needs, surpassing keyword matching and reducing the rate of irrelevant content extraction.

[0098] S22. Using the BERT model to perform semantic encoding on the captured multimodal webpage content to generate a semantic vector of the webpage content;

[0099] S23. Calculate the cosine similarity between the user intention semantic vector and the semantic vector of the webpage content, and filter out valid webpage content related to the user intention based on the similarity score.

[0100] Specifically, this embodiment uses a pre-trained natural language model (BERT) to semantically encode the query statement input by the user, and generates a user intention semantic vector V_query to represent the user's true query intention. Specifically, the user input is segmented and then input into the BERT model for semantic encoding to generate a user intention semantic vector. The web page content including text, picture alt description, video description and other multimodal information is captured through browser automation tools (such as Selenium). On this basis, the system encodes each piece of text in the web page using the BERT model to generate a set of web page content semantic vectors V_content_i. These vectors are all in a unified semantic space, which is convenient for subsequent alignment with user intentions.

[0101] Calculate the semantic similarity (cosine similarity) between V_query and each V_content_i. Based on the similarity score, select webpage paragraphs that closely match the user's intent as candidate content for extraction. This method determines their proximity in semantic space by calculating the cosine of the angle between two vectors. The closer the value is to 1, the more relevant it is.

[0102] Specifically, the system sorts web pages in descending order based on similarity scores. This semantic similarity screening mechanism, acting as a preliminary coarse screening tool, effectively eliminates a large amount of web content irrelevant to the user's intent and constructs a candidate set with semantic proximity to the user's question. This ensures that the subsequent AIGC reverse guidance process is conducted within the scope of semantically credible material, improving the system's overall efficiency and generation accuracy. Through this approach, the system implements a "semantically driven" web content screening strategy, addressing the problem of traditional keyword-based search that easily misses heterogeneous expressions.

[0103] As a further preferred technical solution, in step S20, after screening out valid webpage content related to the user's intention, the method further includes:

[0104] The context window of the filtered valid web page content is expanded, and the web page content structure is reconstructed and returned.

[0105] It should be noted that this embodiment uses a natural language processing model (such as BERT) to semantically model user intent and webpage content, and uses cosine similarity to screen candidate webpage paragraphs that are semantically close to the user query. It also performs operations such as keyword extraction and context expansion to provide high-quality corpus support for subsequent AIGC generation.

[0106] As a further preferred technical solution, in step S30, identifying the keyword encoding in the AIGC-generated content template as the target content semantic vector specifically includes the following steps:

[0107] Use the NER model to extract key semantic units in the content template;

[0108] Combining part-of-speech analysis with the BERT-based attention weighting mechanism, we can identify keywords corresponding to key semantic units from content templates.

[0109] The keywords are encoded into target content semantic vectors.

[0110] This embodiment uses a pre-trained named entity recognition model (BERT-CRF) to identify key semantic units such as proper nouns, functional terms, and technical parameters in the draft text. Specifically, the system sends the draft text into the NER model as an input sequence after word segmentation. The model outputs the corresponding entity label sequence based on the contextual semantic understanding, which is used to identify field-related key phrases such as "hot-dip galvanized steel structure", "corrosion protection level", "220kV voltage level", and "high humidity environment". After the recognition is completed, the above key phrases are extracted from the NER output and a keyword list is constructed. Subsequently, for each phrase in the keyword list, semantic encoding is performed by calling the BERT model: the phrase is passed into the BERT model as input, and the [CLS] bit of its output is used to represent the contextual semantic features of the phrase to obtain the corresponding semantic vector representation.

[0111] Specifically, based on the user's specified and extracted content, AIGC uses advanced deep learning models, such as GPT-3 and DALL-E, to automatically generate relevant text, image or video content. For example, based on the content requirements input by the user (such as "recommend several anti-corrosion towers suitable for 220kV transmission lines in coastal high-humidity areas"), the AIGC model (such as GPT-3) is called to generate a preliminary content draft. For example, AIGC may output: "It is recommended to use hot-dip galvanized steel structure towers or composite insulation material towers. These types have excellent salt spray resistance and are suitable for coastal areas of 220kV single-circuit lines."

[0112] This type of natural language content requires further structuring. The system uses a named entity recognition (NER) model to extract key semantic units such as "tower type," "applicable voltage level," and "usage environment." Combining part-of-speech analysis with a BERT-based attention weighting mechanism, it identifies core fields and keywords within the content, such as "hot-dip galvanized steel structure," "composite materials," "220kV," and "coastal high-humidity environment." These fields are encoded into semantic vectors, forming the target content semantic vector V_target, which serves as the basis for subsequent web content screening and integration.

[0113] As a further preferred technical solution, in step S30, web page content matching the keyword is screened out based on the semantic similarity between the semantic vector of the valid web page content and the semantic vector of the target content, specifically:

[0114] The cosine similarity between the semantic vector of the effective web page content and the semantic vector of the target content is calculated, and the web page content matching the keyword is screened out based on the similarity score.

[0115] Specifically, the multimodal web page content (including text paragraphs, image alt information, etc.) obtained by the aforementioned crawler is uniformly encoded into a web page content semantic vector V_content_i, the semantic similarity between the target content semantic vector V_target and the web page content vector V_content_i is calculated, and the most matching web page fragment is screened out as a supplementary basis.

[0116] As a further preferred technical solution, step S40: fusing the webpage content matching the keyword with the content template to obtain generated content corresponding to the query request, specifically includes:

[0117] Merging webpage content matching the keyword with the content template according to pre-configured fusion rules, wherein the fusion rules include field priority, overlay conditions, and insertion position markers;

[0118] If the webpage content contains more accurate information than the content template, the information is overwritten into the corresponding fragment information in the content template;

[0119] If the web page content includes key information fields that are not included in the content template, the key information fields are inserted into the corresponding positions of the content template.

[0120] Specifically, this embodiment presets field priority labels for key semantic fields in the content template (such as "tower type", "applicable voltage level", "environmental conditions", etc.). When the system detects during the fusion process that the same field has different values ​​in the AIGC draft and the web page extracted content (for example, the "tower material" in the draft is "aluminum alloy structure", while the web page content is "hot-dip galvanized steel structure"), it determines that there is a content conflict in the field. At this time, automatic judgment and processing are performed based on the predefined field priority rules: if the field priority is set to "web data priority", the content extracted from the web page is retained and the AIGC generated value is overwritten; if it is set to "draft priority", the original AIGC generated content is retained. The priority setting can be flexibly configured based on the professionalism, structured degree or user-set preferences of the field to achieve reliability and expression accuracy of the fusion results.

[0121] It should be noted that in order to ensure the accuracy and professionalism of the content generated by AIGC, a rule-based fusion strategy is adopted for text integration, which mainly includes: replacement mechanism: if the web page content contains more precise and authoritative terms or data information (such as standard numbers, specific models), the system will replace the corresponding fragments in AIGC with web page content; insertion mechanism: if the web page information provides key information fields not covered by the draft (such as applicable wind speed range, material grade), it will be inserted into the corresponding position of the draft.

[0122] It should be noted that the above fusion rules are defined by the system's preset policy configuration file, which includes parameters such as field priority, overwrite conditions, and insertion position markers. The fusion process is driven by the rules engine at the program level and does not require manual user intervention.

[0123] As a further preferred technical solution, after the step S40 of fusing the webpage content matching the keyword with the content template to obtain generated content corresponding to the query request, the method further includes:

[0124] The BART model is called to perform language polishing on the generated content to obtain the final generated content.

[0125] It should be noted that, after content fusion, this embodiment calls the language model (BART) to perform language polishing to improve the fluency and professionalism of the text.

[0126] As a further preferred technical solution, this embodiment uses a high-performance database system, such as Elasticsearch, to store the extracted and generated content, and can design an efficient indexing strategy to support fast data retrieval and update, ensuring that the system can handle the storage and access needs of large-scale data.

[0127] As a further preferred technical solution, in order to further enhance the user experience, this embodiment designs an intuitive user interface through which users can directly interact with the crawler and content generation system and customize the parameters for content extraction and generation. Users can intuitively see the process and results of content extraction and generation through visualization tools such as charts and sliders, enhancing their sense of control and participation. Through this visual interface, users can not only view the generated content, but also adjust the generated parameters such as content theme, style, format, etc. according to their personal needs. This interactive method can enhance user participation and make content generation more closely aligned with user needs.

[0128] As a further preferred technical solution, this embodiment introduces a feedback mechanism that allows users to evaluate the extracted and generated content. User feedback (such as click-through rate, likes, comments, etc.) will be fed back to the system in real time, and the data will be used to adjust and optimize the algorithm, so that the content extraction and generation process can adaptively learn and continuously improve. In this way, the system can not only optimize the generation strategy according to the specific needs of users, but also adjust the crawler's crawling strategy based on user feedback, improving the accuracy and quality of content in real time.

[0129] In practice, user preferences and needs for content change over time. Therefore, the adaptive learning mechanism employed by this invention continuously tracks and optimizes user interests, making the system's content recommendations and generation increasingly precise. For example, users of an e-commerce platform may develop varying interests in clothing styles and brands as the seasons change. The system will automatically adjust its content generation strategy to generate customized product recommendations that meet current user needs.

[0130] In addition, if Figure 2 As shown, the second embodiment of the present invention further proposes a content extraction and AIGC fusion generation system, which includes:

[0131] The first response module 10 is used to respond to the query request input by the user and crawl the web page content using crawler technology;

[0132] The second response module 20 is used to respond to the query request input by the user and generate a content template using AIGC;

[0133] A primary screening module 30 is configured to screen out valid web page content related to the user's intent based on the semantic similarity between the user's intent semantic vector corresponding to the query request and the semantic vector of the web page content;

[0134] Secondary screening module 40, for identifying keywords encoded in the AIGC-generated content template as target content semantic vectors, and screening out web page content that matches the keywords based on the semantic similarity between the semantic vectors of valid web page content and the semantic vectors of the target content;

[0135] The fusion module 50 fuses the webpage content that matches the keyword with the content template to obtain generated content corresponding to the query request.

[0136] As a further preferred technical solution, the first response module 10 is used to set a starting URL (seed URL) in a Scrapy configuration file so that the crawler can start crawling data from a specified website.

[0137] Scrapy's scheduling system automatically controls the crawler's crawling progress, ensuring that the crawled content does not exceed the predetermined scope. After crawling the target page, the custom crawler needs to process the crawled content. For static web pages, the crawler uses the requests library to crawl the page and parses the page with BeautifulSoup to extract the text, images, and video content. The crawler automatically extracts the required data from the HTML code, such as product descriptions, news content, or reviews. For dynamic web pages (such as content dynamically loaded via JavaScript or AJAX), the custom crawler uses Selenium to simulate browser behavior, executing JavaScript code on the page and loading dynamic data. Selenium allows the crawler to simulate user actions such as clicks and scrolling to retrieve JavaScript-rendered content. This allows the crawler to crawl not only static content but also dynamically generated data (such as real-time prices and reviews). Some web pages require user interaction to load content, such as "Load More" buttons or pagination buttons on e-commerce websites. The custom crawler simulates user clicks using Selenium or Playwright, automatically loading more content and crawling this data. The crawler can simulate actions such as clicking buttons and scrolling pages on the page to trigger JavaScript code and obtain more data.

[0138] It should be noted that the captured web page content needs to be extracted and stored as structured data to facilitate subsequent analysis and use. The custom crawler parses the HTML content through BeautifulSoup, extracts the captured text data and converts it into a structured format (such as JSON or CSV). For images, the crawler can extract the URL of the image and download the image using the requests library. For videos, the crawler can extract the embed code or URL address of the video and store it as structured data. At the same time, the crawler will clean and deduplicate the captured data. Through the pandas library, the crawler can easily remove irrelevant content (such as advertisements, page navigation menus, etc.) and format the data into a unified form to ensure data consistency and accuracy. For example, information such as date format and price format will be standardized in this process.

[0139] It should be noted that the crawled structured data will be stored in MongoDB or Elasticsearch to ensure efficient data retrieval and query. MongoDB is a NoSQL database suitable for storing large amounts of unstructured data, while Elasticsearch is suitable for full-text search and indexing, facilitating fast data retrieval and query.

[0140] In this embodiment, the crawler accesses the target web page according to the aforementioned predetermined rules and algorithms to capture the required data. During this process, the crawler analyzes the web page's DOM structure and extracts valuable information such as text, images, and video links. To process dynamic content, the crawler may execute JavaScript or simulate user interaction to access content dynamically generated by scripts.

[0141] As a further preferred technical solution, the first response module 10 further includes:

[0142] The strategy update unit is used to establish a web page structure template based on the DOM tree when visiting a web page for the first time; and to compare the structure of the currently visited page with the web page structure template when visiting the web page subsequently, and to update the crawling strategy when it is determined that the difference in the web page structure exceeds a set threshold.

[0143] As a further preferred technical solution, the updating process of the crawling strategy includes:

[0144] Update matching rules and field paths, and retrain the field recognition model;

[0145] Combined with user feedback indicators, dynamically adjust crawling frequency, field priority, and content area positioning strategies.

[0146] As a further preferred technical solution, the primary screening module 30 specifically includes:

[0147] A user intent semantic vector generation unit is used to semantically encode the query request input by the user using a pre-trained BERT model to generate a user intent semantic vector;

[0148] A semantic vector generation unit for web page content, configured to perform semantic encoding on the captured multimodal web page content using the BERT model to generate a semantic vector for the web page content;

[0149] The first similarity calculation unit is used to calculate the cosine similarity between the user intention semantic vector and the semantic vector of the web page content, and to screen out valid web page content related to the user intention based on the similarity score.

[0150] As a further preferred technical solution, the primary screening module 30 further includes:

[0151] The content expansion unit is used to expand the context window of the filtered valid web page content and reconstruct the web page content structure before returning it.

[0152] As a further preferred technical solution, the primary screening module 40 specifically includes:

[0153] A target content semantic vector generation unit is used to extract key semantic units in the content template using the NER model; combine part-of-speech analysis and the attention weight mechanism based on the BERT model to identify keywords corresponding to the key semantic units from the content template; and encode the keywords into a target content semantic vector;

[0154] The second similarity calculation unit is used to calculate the cosine similarity between the semantic vector of the effective web page content and the semantic vector of the target content, and to screen out the web page content matching the keyword according to the similarity score.

[0155] As a further preferred technical solution, the fusion module 50 is used to fuse the webpage content matching the keyword with the content template according to pre-configured fusion rules, wherein the fusion rules include field priority, coverage conditions, and insertion position markers;

[0156] If the webpage content contains more accurate information than the content template, the information is overwritten into the corresponding fragment information in the content template;

[0157] If the web page content includes key information fields that are not included in the content template, the key information fields are inserted into the corresponding positions of the content template.

[0158] As a further preferred technical solution, the system further includes a content optimization module, which is specifically used to: call the BART model to perform language polishing on the generated content to obtain the final generated content.

[0159] It should be noted that other embodiments or specific implementation methods of the content extraction and AIGC fusion generation system of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0160] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0161] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0162] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0163] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0164] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for content extraction and AIGC fusion generation, characterized in that: include: Respond to user input queries, use crawler technology to crawl web content, and use AIGC to generate content templates; Filter out valid web page content related to the user's intent based on the semantic similarity between the user's intent semantic vector corresponding to the query request and the semantic vector of the web page content; Identify keywords in the AIGC-generated content template and encode them into target content semantic vectors. Filter out web page content that matches the keywords based on the semantic similarity between the semantic vectors of valid web page content and the semantic vectors of target content. The web page content that matches the keyword is integrated with the content template to obtain the generated content corresponding to the query request.

2. The content extraction and AIGC fusion generation method according to claim 1, characterized in that: The crawling technology used to crawl web page content includes: When a web page is first visited, a web page structure template is created based on the DOM tree; When accessing the web page subsequently, the structure of the currently accessed page is compared with the web page structure template, and the crawling strategy is updated when it is determined that the web page structure difference exceeds the set threshold.

3. The content extraction and AIGC fusion generation method according to claim 2, characterized in that: The crawl strategy update process includes: Update matching rules and field paths, and retrain the field recognition model; Combined with user feedback indicators, dynamically adjust crawling frequency, field priority, and content area positioning strategies.

4. The method for content extraction and AIGC fusion generation according to claim 1, wherein: The method of screening out valid web page content related to the user intent based on the semantic similarity between the user intent semantic vector corresponding to the query request and the semantic vector of the web page content includes: Use the pre-trained BERT model to semantically encode the query request entered by the user and generate the user intent semantic vector; Perform semantic encoding on the captured multimodal web page content using the BERT model to generate a semantic vector for the web page content; The cosine similarity between the user intention semantic vector and the semantic vector of the web page content is calculated, and the valid web page content related to the user intention is filtered out based on the similarity score.

5. The method for content extraction and AIGC fusion generation according to claim 4, characterized in that: After filtering out valid webpage content related to the user's intention, the method further includes: The context window of the filtered valid web page content is expanded, and the web page content structure is reconstructed and returned.

6. The method for content extraction and AIGC fusion generation according to claim 1, characterized in that: The identification of keywords in the AIGC-generated content template and encoding them into target content semantic vectors includes: Use the NER model to extract key semantic units in the content template; Combining part-of-speech analysis with the BERT-based attention weighting mechanism, we can identify keywords corresponding to key semantic units from content templates. The keywords are encoded into target content semantic vectors.

7. The method for content extraction and AIGC fusion generation according to claim 1, characterized in that: The step of screening out web page content that matches the keyword based on the semantic similarity between the semantic vector of the valid web page content and the semantic vector of the target content includes: The cosine similarity between the semantic vector of the effective web page content and the semantic vector of the target content is calculated, and the web page content matching the keyword is screened out based on the similarity score.

8. The method for content extraction and AIGC fusion generation according to claim 1, wherein: The step of fusing the webpage content matching the keyword with the content template to obtain generated content corresponding to the query request includes: Merging webpage content matching the keyword with the content template according to pre-configured fusion rules, wherein the fusion rules include field priority, overlay conditions, and insertion position markers; If the webpage content contains more accurate information than the content template, the information is overwritten into the corresponding fragment information in the content template; If the web page content includes key information fields that are not included in the content template, the key information fields are inserted into the corresponding positions of the content template.

9. The method for content extraction and AIGC fusion generation according to claim 8, characterized in that: After fusing the webpage content matching the keyword with the content template to obtain generated content corresponding to the query request, the method further includes: The BART model is called to perform language polishing on the generated content to obtain the final generated content.

10. A content extraction and AIGC fusion generation system, characterized in that: include: The first response module is used to respond to the query request input by the user and crawl the web page content using crawler technology; The second response module is used to respond to the query request input by the user and generate a content template using AIGC; A primary screening module is used to screen out valid web page content related to the user's intent based on the semantic similarity between the user's intent semantic vector corresponding to the query request and the semantic vector of the web page content; The secondary screening module is used to identify keywords in the content template generated by AIGC and encode them into target content semantic vectors. Based on the semantic similarity between the semantic vectors of valid web page content and the semantic vectors of target content, the module selects web page content that matches the keywords. The fusion module fuses the web page content that matches the keyword with the content template to obtain the generated content corresponding to the query request.

Images