Method and device for achieving intelligent AI webpage content obtaining based on LLM, storage medium and program product

Abstract

The invention discloses a method and device for achieving intelligent AI webpage content obtaining based on LLM, a storage medium and a program product, and relates to the technical field of data processing. According to the method, the search instruction input by the user is received, the search instruction is input into the trained semantic analysis model, after the semantic analysis model converts the search instruction into the query parameter, the query parameter is sent to the search engine, then the webpage obtained by the search engine according to the query parameter is received, and the search result is obtained. The method comprises the following steps: acquiring a webpage, determining a target webpage from the webpage according to a preset screening rule, crawling an HTML (Hypertext Markup Language) source code of the target webpage based on a pre-acquired authorization result, inputting the HTML source code into a trained LLM model, and determining a target code label in the LLM model according to the HTML source code to obtain text content in the target code label. Wherein the LLM model has a strong language understanding capability and can dynamically adapt to the change of a webpage structure, so that the time cost of crawling webpage contents is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, storage medium and program product for implementing intelligent AI to obtain web page content based on LLM. Background Art

[0002] In the traditional method of acquiring web page data, a traditional crawler is first used to automatically access web pages and download their Hypertext Markup Language (HTML) content based on a predefined list of Uniform Resource Locators (URLs) or seed URLs. Subsequently, the HTML page is parsed with the help of regular expressions or Extensible Markup Language Path Language (XPath) to locate and extract the required data fields contained in specific elements in the page.

[0003] However, the above methods mainly rely on preset rules or patterns to extract web page data, but lack the ability to deeply understand and semantically analyze web page content. This results in the crawler program being difficult to adapt to changes in the web page structure when facing complex or non-standardized web pages, which in turn causes data extraction failure or inaccurate data extraction. At this time, complex conversion logic and manual intervention are often required to continuously update and maintain crawler rules to adapt to changes in web pages, resulting in a high time cost for web page content acquisition.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a method, device, storage medium and program product for realizing intelligent AI acquisition of web page content based on LLM, aiming to solve the technical problem of high time cost of acquiring web page content.

[0006] To achieve the above objectives, the present application proposes a method for realizing intelligent AI acquisition of webpage content based on LLM, wherein the method comprises: Receive a search instruction input by a user, input the search instruction into a trained semantic parsing model, and after the semantic parsing model converts the search instruction into a query parameter, send the query parameter to a search engine; Receiving web pages acquired by the search engine according to the query parameters, and determining a target web page from the web pages according to a preset screening rule; Based on the authorization result obtained in advance, the HTML source code of the target webpage is crawled, the HTML source code is input into the trained LLM model, and after the LLM model determines the target code tag according to the HTML source code, the text content in the target code tag is obtained.

[0007] In one embodiment, the semantic parsing model receives the search instruction and performs word segmentation on the search instruction to obtain at least one word; The semantic parsing model identifies entity words in the search instruction according to a preset named entity recognition algorithm, and uses the entity words as the query parameters.

[0008] In one embodiment, the step of determining the target web page from the web pages according to the preset screening rule includes: Obtaining a preset screening element, wherein the screening element includes the number of times the query parameter appears in the webpage, the update time of the webpage, and the source of the webpage; Determine the screening element score corresponding to each of the web pages based on the preset element weights, and perform weighted summation on the screening element scores to obtain the score of each of the web pages; The web page with a score greater than a preset threshold is used as the target web page.

[0009] In one embodiment, the step of determining the target code tag according to the HTML source code by the LLM model includes: Receiving the HTML source code, parsing the HTML source code into a DOM tree, and constructing tag relationship graph data according to the DOM tree; According to the preset internal cluster compactness, hierarchical clustering analysis is performed on the tag nodes in the DOM tree to obtain tag hierarchical clustering data; According to the label relationship graph data and the label hierarchical clustering data, matching is performed in a preset knowledge base to determine the webpage type of the target webpage; Based on the webpage type of the target webpage, a target code tag in the HTML source code is determined.

[0010] In one embodiment, the step of determining the target code tag in the HTML source code based on the webpage type of the target webpage includes: Determining a valid code tag corresponding to the webpage type according to the preset knowledge base; Based on the effective code tag, the target code tag is determined in the DOM tree.

[0011] In one embodiment, the step of determining the target code tag in the DOM tree based on the valid code tag includes: Read the first category name corresponding to the code tag in the DOM tree, and the second category name corresponding to the valid code tag; Convert the first category name and the second category name into an embedding vector, and calculate the similarity between the first category name and the second category name; If the similarity between the first category name and the second category name is greater than a preset threshold, the code label corresponding to the first category name is determined to be the target code label.

[0012] In one embodiment, the step of extracting the text content in the target code tag includes: Traverse the child nodes of the target code tag node in the DOM tree; Access the node type attribute of the child node; If the node type attribute of the child node is a text node, the text content corresponding to the child node is extracted.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a device for implementing intelligent AI to obtain web page content based on LLM, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the method for implementing intelligent AI to obtain web page content based on LLM as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the method for realizing intelligent AI to obtain web page content based on LLM as described above are implemented.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the method for realizing intelligent AI to obtain web page content based on LLM as described above.

[0016] The present application provides a method, device, storage medium and program product for realizing intelligent AI acquisition of web page content based on LLM, by receiving a search instruction input by a user, inputting the search instruction into a trained semantic parsing model, and after the semantic parsing model converts the search instruction into a query parameter, sending the query parameter to the search engine, and then receiving the web page obtained by the search engine according to the query parameter, and determining the target web page from the web page according to the preset screening rules, obtaining the HTML source code of the target web page based on the pre-acquired authorization result, inputting the HTML source code into a trained LLM model, and obtaining the text content in the target code tag after the LLM model determines the target code tag according to the HTML source code. The method analyzes the HTML source code of the target web page through a trained LLM model, and can automatically identify the target code tag and its content, wherein the LLM model has a strong language understanding ability, can dynamically adapt to changes in the structure of the web page, does not need to rely on fixed rules or patterns, and reduces the time cost of manually modifying the crawler code to obtain web page content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A flowchart diagram of the first embodiment of the method for implementing intelligent AI to obtain webpage content based on LLM in this application; Figure 2 A flowchart diagram of the second embodiment of the method for implementing intelligent AI to obtain webpage content based on LLM in this application; Figure 3 A flowchart diagram of the third embodiment of the method for implementing intelligent AI to obtain webpage content based on LLM in this application; Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the method for implementing intelligent AI to obtain web page content based on LLM in an embodiment of the present application.

[0020] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] In the traditional method of acquiring web page data, a traditional crawler is first used to automatically access web pages and download their Hypertext Markup Language (HTML) content based on a predefined list of Uniform Resource Locators (URLs) or seed URLs. Subsequently, the HTML page is parsed with the help of regular expressions or Extensible Markup Language Path Language (XPath) to locate and extract the required data fields contained in specific elements in the page.

[0024] However, the above methods mainly rely on preset rules or patterns to extract web page data, but lack the ability to deeply understand and semantically analyze web page content. This results in the crawler program being difficult to adapt to changes in the web page structure when facing complex or non-standardized web pages, which in turn causes data extraction failure or inaccurate data extraction. At this time, complex conversion logic and manual intervention are often required to continuously update and maintain crawler rules to adapt to changes in web pages, resulting in a high time cost for web page content acquisition.

[0025] In view of the above problems, this application proposes a method for realizing intelligent AI acquisition of web page content based on LLM (Large Language Model), which receives the search instruction input by the user, inputs the search instruction into a trained semantic parsing model, and after the semantic parsing model converts the search instruction into a query parameter, sends the query parameter to the search engine, and then receives the web page obtained by the search engine according to the query parameter, and determines the target web page from the web page according to the preset screening rules, crawls the HTML source code of the target web page based on the pre-acquired authorization result, inputs the HTML source code into the trained LLM model, and obtains the text content in the target code tag after the LLM model determines the target code tag according to the HTML source code. This method analyzes the HTML source code of the target web page through the trained LLM model, and can automatically identify the target code tag and its content. Among them, the LLM model has a strong language understanding ability, can dynamically adapt to the changes in the structure of the web page, does not need to rely on fixed rules or patterns, and reduces the time cost of manually modifying the crawler code to obtain web page content.

[0026] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, etc., or an electronic device capable of realizing the above functions, etc. The following takes the webpage content acquisition artificial intelligence (AI, Artificial Intelligence) system based on the LLM model as an example to illustrate this embodiment and the following embodiments.

[0027] Based on this, the first embodiment proposed in this application provides a method for realizing intelligent AI acquisition of webpage content based on LLM, referring to Figure 1 In this embodiment, the method for implementing intelligent AI to obtain webpage content based on LLM includes steps S10 to S30: Step S10, receiving a search instruction input by a user, inputting the search instruction into a trained semantic parsing model, and after the semantic parsing model converts the search instruction into query parameters, sending the query parameters to a search engine.

[0028] In the case of obtaining website authorization and desensitizing privacy data, traditional crawler programs usually rely on fixed URL lists or rules to crawl web pages, lacking in-depth understanding of user search intent and semantic analysis capabilities. This embodiment introduces a semantic parsing model to achieve intelligent parsing and conversion of user search instructions, allowing crawler programs to more flexibly adapt to different search engines and complex web page structures, and improve the intelligence level of crawling web page content.

[0029] Exemplarily, a user inputs a search command through a user interface or interface, and the semantic parsing model performs deep semantic analysis on the command input by the user, understands the user's true intention, and converts this intention into query parameters that the computer can understand. The query parameters are then sent to the search engine, and the search engine performs the search task based on these parameters and returns relevant search results.

[0030] Optionally, a user interface is set up to provide a text input box in which the user can enter a search instruction, or the user is allowed to enter a search instruction through a microphone, and the user's voice is converted into text using a speech recognition algorithm, which is then used as the search instruction.

[0031] Optionally, step S10 includes steps S11-S12: Step S11: the semantic parsing model receives the search instruction and performs word segmentation on the search instruction to obtain at least one word.

[0032] Step S12: the semantic parsing model identifies entity words in the search instruction according to a preset named entity recognition algorithm, and uses the entity words as the query parameters.

[0033] Before inputting the search instruction into the semantic parsing model, the search instruction can be preprocessed, including removing irrelevant characters, word segmentation, part-of-speech tagging, etc., so that the semantic parsing model can better understand and process the search instruction. The preprocessed search instruction is input into the trained semantic parsing model, and the semantic parsing model uses deep learning technologies such as long short-term memory network (LSTM), bidirectional Transformer encoder representation (BERT, BidirectionalEncoder Representations from Transformers), etc. to deeply understand and analyze the search instruction. Among them, the model based on deep learning can automatically learn the deep features and contextual information of words, so as to more accurately segment words and obtain a word list. Afterwards, the semantic parsing model extracts the text representation of the word, the contextual information features such as the previous and next words, etc. according to the preset named entity recognition algorithm such as hidden Markov model (HMM, Hidden Markov Model), conditional random field (CRF, Conditional Random Field), etc., and classifies each word in the word list to determine whether it belongs to an entity word. By identifying the entity words in the sentence, the query parameters are extracted. For example, from “I want to read the latest technology news”, the semantic parsing model can recognize “technology” and “news” as query parameters.

[0034] Step S20, receiving web pages acquired by the search engine according to the query parameters, and determining a target web page from the web pages according to a preset screening rule.

[0035] After receiving the query parameters, the search engine will search for web pages related to the query parameters in the web page database according to its internal index and ranking algorithm. After the search is completed, the web pages that match the query parameters are returned to the web page content acquisition artificial intelligence system based on the LLM model. After receiving the web pages returned by the search engine, the web page content acquisition artificial intelligence system based on the LLM model further screens these web pages according to specific needs or goals. In order to achieve this screening process, screening rules can be preset in advance. The screening rules can be based on multiple dimensions such as the content, source, release time, relevance score, user evaluation, etc. of the web page. According to these preset screening rules, the web page content acquisition artificial intelligence system based on the LLM model will evaluate the received list of web pages one by one, and finally determine the web pages that meet the requirements as the target web pages.

[0036] Optionally, step S20 includes steps S21 to S23: Step S21, obtaining preset screening elements, wherein the screening elements include the number of times the query parameter appears in the web page, the update time of the web page, and the source of the web page.

[0037] The screening element is an indicator used to evaluate whether a web page meets specific conditions. In this embodiment, the screening elements include but are not limited to the number of times the query parameter appears in the web page, the update time of the web page, and the source of the web page. For each screening element, the web page content acquisition artificial intelligence system based on the LLM model will obtain its corresponding value. For example, a text matching algorithm is used to calculate the frequency of query parameters appearing in the web page content; the update time of the web page is extracted from the metadata or HTML tags of the web page; the source of the website is determined based on the domain name, URL structure, or a known authoritative website list of the web page.

[0038] Exemplarily, a screenshot tool is used to take a screenshot of a web page, and the text content in the screenshot is identified. The text content is segmented into independent words or phrases using a word segmentation tool such as a Chinese word segmenter, and then the number of times the query parameter appears is counted. The screening elements may also include the correlation between the query parameter and the web page title. For example, a crawler program is used to first crawl the title of the web page, and a semantic matching algorithm such as word vector (word2vec, word to vector), bidirectional Transformer encoder representation, etc. is used to calculate the correlation between the query parameter and the web page title. For the update time of the web page, the HTML code of the web page can be parsed by an HTML parser to find tags containing update time information, such as " <meta> " tag, "Last-Modified" attribute in the "" part, etc. Extract metadata related to the update time from the parsed HTML. As for the source of the web page, the URL in the web page can be parsed through regular expressions or URL parsing libraries, and the website source of the web page can be determined based on the URL.

[0039] Step S22, determining the screening element score corresponding to each of the web pages based on the preset element weights, and performing weighted summation on the screening element scores to obtain the score of each of the web pages.

[0040] Step S23: taking the webpage whose score is greater than a preset threshold as the target webpage.

[0041] Exemplarily, an element weight is determined for the screening element, and the screening element score corresponding to the web page is calculated according to the element weight. Among them, the more times the query parameter appears in the web page, and the closer the update time of the web page is to the query time, the more valuable the web page content is, and the larger the corresponding screening element score is. A preset website list is established inside the web page content acquisition artificial intelligence system based on the LLM model. The website list is sorted according to the authority of the website, and the sorted website list is used as the basis for the subsequent screening element score. Different intervals are divided into different screening element scores for the website list. If the web page source is located at the front of the list, a higher screening element score is assigned. If the web page source is not in the website list, a lower screening element score is assigned. Finally, the score of each screening element is multiplied by its corresponding element weight, and then weighted summed to obtain the total score of each web page, and the web page with a score greater than the preset threshold is used as the target web page.

[0042] Step S30, based on the authorization result obtained in advance, crawl the HTML source code of the target web page, input the HTML source code into the trained LLM model, and after the LLM model determines the target code tag according to the HTML source code, extract the text content in the target code tag.

[0043] After obtaining website authorization and desensitizing private data, the crawler program is used to capture the HTML source code of the target web page, and the HTML source code is passed as input data to a trained LLM model. The LLM model will analyze the input HTML source code, identify the various HTML tags in it, and determine the target code tags from which text content needs to be extracted based on the trained algorithm logic. Once the target code tags are determined, the LLM model will extract text content from these tags, including web page titles, paragraph text, link text, etc.

[0044] The LLM model determines the web page type of the target web page according to the HTML source code, and determines the target code tag in the HTML source code according to the web page type.

[0045] It should be noted that different types of web pages have different structures. For example, news web pages usually have a title, text, author, release date, etc., product web pages usually have a product name, price, specifications, evaluation, etc., and forum web pages usually have a topic, author, release date, comments, etc. The LLM model can infer the type of web page based on its structure. Different types of web pages also use different HTML tags. For example, news web pages usually use " <h1>" tag indicates the title, and product pages usually use " "Tags indicate product information, and forum pages usually use " " tag indicates a user comment. Therefore, compared with the traditional method of extracting web page data through preset regular expressions or XPath, the method of using the LLM model to analyze HTML source code and crawling text content according to the type of web page in this embodiment can automatically adapt to web pages with complex or non-standard structures without manually updating the crawler program code, which can reduce the time cost of crawling web page content.

[0046] Specifically, we first collect a large number of different types of web page HTML source codes, annotate these HTML source codes, and clarify the type of each web page and the HTML tags in each type of web page. We use the collected annotated data to train the LLM model, learn how to identify the type of web page from the HTML source code, and understand the purpose of HTML tags in different types of web pages. When a new web page HTML source code is input into the LLM model, the LLM model will first analyze the structure and content of the HTML web page, use the knowledge it has learned during the training process to infer the type of the web page, and then identify the corresponding target code tag according to the web page type, and extract the text content from the target code tag.

[0047] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 In step S30, the step of determining the target code tag according to the HTML source code by the LLM model includes steps S31 to S34: Step S31, receiving the HTML source code, parsing the HTML source code into a DOM tree, and constructing tag relationship graph data according to the DOM tree.

[0048] Exemplarily, after receiving the HTML source code, the LLM model parses the HTML source code into a DOM (Document Object Model) tree through an HTML parser. The DOM tree is a tree structure in which each node represents an element, attribute or text node in the HTML document. Based on the DOM tree, the tag relationship graph data is constructed. The nodes in the tag relationship graph data represent tags in the DOM tree, such as " ”、" ”、" ", etc. The edges represent the parent-child relationship, sibling relationship or other custom relationship between tags. Parsing HTML source code into a DOM tree helps to understand the structure and content of a web page, and the tag relationship graph data can more intuitively represent the relationship between tags.

[0049] Specifically, the LLM model first traverses the DOM tree, starting from the root node using algorithms such as depth-first search or breadth-first search. For each tag node in the DOM tree, a corresponding graph node is created in the graph structure, and other related content in the tag is associated with the graph node as child nodes. After traversing the DOM tree and adding all necessary edges, the constructed tag relationship graph data is output.

[0050] Step S32, performing hierarchical clustering analysis on the tag nodes in the DOM tree according to the preset internal cluster compactness, to obtain tag hierarchical clustering data.

[0051] It should be noted that hierarchical cluster analysis is an unsupervised learning algorithm that can divide a data set into multiple levels or clusters that have similar characteristics under specific metrics, while the internal cluster closeness is used to determine the closeness of the label nodes within each cluster in the labeled hierarchical clustering data.

[0052] Exemplarily, a unique cluster identifier is assigned to each tag node in the DOM tree, the similarity or distance between each pair of clusters is calculated, and the cluster pair with the highest similarity or the smallest distance is selected for merging. The internal cluster compactness of the merged tag hierarchical clustering data is calculated, and when the internal compactness of the merged cluster reaches or exceeds a preset threshold, the merging is stopped. For example, in a news webpage, " <header> " tag is usually used to contain introductory content of the page, including " <h1>”、"< / h1> <h2>”、"< / h2> <h3>" labels, representing the first-level title, second-level title, and third-level title respectively. Through label hierarchical clustering, these titles can be merged into one cluster to more clearly show the hierarchical structure between labels.

[0053] The internal cluster density can be calculated based on the distance from the label node within the cluster to the cluster center, such as the average square error or the average Manhattan distance, and the cluster center can be customized by the user. Tag-level clustering data can more clearly show the hierarchical relationship between the tags in the HTML source code, thereby reflecting the structure of the web page.

[0054] Step S33, matching the tag relationship graph data and the tag hierarchical clustering data in a preset knowledge base to determine the webpage type of the target webpage.

[0055] Exemplarily, a knowledge base is constructed in the LLM model, covering multiple web page types and the label relationship graph and hierarchical clustering features corresponding to each web page type. A specific similarity calculation method such as cosine similarity is used to compare the label relationship graph data and label hierarchical clustering data of the target web page with the label relationship graph and hierarchical clustering features corresponding to each web page type in the knowledge base one by one, and a similarity score is calculated. According to the similarity score, the web page type with the highest score is selected as the matching result output.

[0056] Step S34: determining the target code tag in the HTML source code based on the webpage type of the target webpage.

[0057] It should be noted that different types of web pages have different structural and content characteristics, so the corresponding HTML tags will also be different. Determining the target code tag based on the type of web page can more accurately locate the required HTML elements, thereby improving the accuracy of data extraction or web page parsing. For example, for news web pages, you can focus on extracting key information such as title, text, and release time; while for e-commerce web pages, you need to pay attention to product names, prices, pictures, and purchase links. Different types of web pages have different information display requirements and layout styles, so different HTML tags are used to achieve them. For example, news web pages usually use "< / h3> <h1>”、"< / h1> <h2>" to define the title, while e-commerce pages use " ”、" " and other tags to layout product information and purchase buttons.

[0058] Exemplarily, the LLM model is trained to identify the HTML tags corresponding to the web page content in different types of web pages. First, a large amount of different types of web page data is collected, including news, e-commerce, blogs, etc. The web page data contains HTML source code and corresponding tag information. Key features are manually annotated from the HTML source code, including tag names, tag attributes, tag hierarchical relationships, text content, etc. These web page data are divided into training sets and validation sets, and appropriate machine learning or deep learning algorithms such as support vector machines, random forests, neural networks, etc. are selected. The LLM model is trained using the training set. During the training process, the LLM model will learn how to predict output tags based on the input HTML source code. After that, the validation set is used to evaluate the performance of the model through cross-validation or other evaluation methods, and the model is optimized based on the evaluation results, including adjusting model parameters, increasing the number of features, or improving feature extraction methods.

[0059] Optionally, step S34 includes steps S341-S342: Step S341, determining a valid code tag corresponding to the webpage type according to the preset knowledge base.

[0060] Exemplarily, in the knowledge base within the LLM model, for each web page type, valid code tags for identifying text content in the web page are listed. For example, a news web page may focus on " <title>”、"< / title> <h1>"After determining the webpage type of the target webpage, the LLM model retrieves a list of valid code tags related to the type from the preset knowledge base.

[0061] Step S342: determining the target code tag in the DOM tree based on the valid code tag.

[0062] Exemplarily, the DOM tree is traversed, and according to the valid code tag list obtained from the knowledge base, the nodes of the DOM tree are recursively traversed to check whether the tag name of each node matches the tag in the list.

[0063] Optionally, step S342 includes steps S3421 to S3423: Step S3421, reading the first category name corresponding to the code tag in the DOM tree, and the second category name corresponding to the valid code tag.

[0064] Step S3422: convert the first category name and the second category name into embedding vectors, and calculate the similarity between the first category name and the second category name.

[0065] Step S3423: If the similarity between the first category name and the second category name is greater than a preset threshold, the code tag corresponding to the first category name is determined to be the target code tag.

[0066] It should be noted that in HTML, a class name is an attribute that can be attached to an HTML element such as an HTML code tag to specify the style or behavior of the element. The class name uses "class" as the attribute name, and its value is one or more class identifiers separated by spaces. For example: This is a header ",in," The " element (i.e., code tag) has two class names: "header" and "main-header". In web design and development, developers usually follow certain naming conventions to imply or indicate the content, function, or style of an element through the class name. For example, a class name "product-description" is likely to indicate that the element contains product description information. By analyzing the class name, the semantics of the text content within the code tag can be identified and derived.

[0067] In the traditional method of acquiring web page data, when parsing HTML pages through regular expressions or XPath to locate and extract code tags in the page, a hard-coded method is usually used to query tags with specific sub-elements or attributes. However, when the structure of the HTML page changes, or the developer changes the naming of the class name, the originally valid regular expression or XPath expression will become invalid, and the content in the web page cannot be acquired. Therefore, the semantic information of the text content in the code tag can be analyzed through the class name of the code tag to determine whether a code tag is a target code tag.

[0068] Exemplarily, the DOM tree is traversed to find the class attribute value of each code tag, i.e., the first class name, and at the same time, valid code tags and their corresponding expected class attribute values, i.e., the second class name, are obtained from the preset knowledge base. By comparing the actual class attribute value of the code tag with the expected value, it can be determined whether the tag meets the conditions of the target code tag.

[0069] Specifically, the first category name and the second category name are converted into embedding vectors using a pre-trained word embedding model. Then, the similarity between these embedding vectors is calculated. The embedding vector can capture the semantic relationship between the category names. By calculating the similarity, the semantic proximity of the two names can be evaluated. The calculated similarity is compared with a preset threshold. If the similarity is greater than the preset threshold, the code label corresponding to the first category name is considered to be the target code label.

[0070] In this embodiment, the HTML source code is parsed by the LLM model, a tag relationship graph and hierarchical clustering data are constructed, and the web page type is matched in combination with the preset knowledge base, thereby determining the target code tag. Specifically, the model first parses the HTML source code into a DOM tree and constructs a tag relationship graph to clearly display the parent-child, brother and other relationships between tags; then the tag nodes are hierarchically clustered according to the preset internal cluster density to divide the tag clusters. Subsequently, the target web page type is matched in the knowledge base in combination with the tag relationship graph and the hierarchical clustering data. Through hierarchical clustering and the tag relationship graph, the web page structure can be more clearly reflected and the target code tag can be accurately located. Finally, according to the web page type, the valid code tag is obtained from the knowledge base, and the target code tag is further determined in the DOM tree through the embedding vector similarity of the class name. Based on the semantic analysis of the class name, the parsing failure caused by the change of HTML structure or class name can be avoided, thereby improving the flexibility of web page content extraction and reducing the time cost of web page content acquisition.

[0071] Based on the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 3 , step S30 also includes steps S35-S36: Step S35, traversing the child nodes of the target code tag node in the DOM tree.

[0072] Step S36, accessing the node type attribute of the child node.

[0073] Step S37: If the node type attribute of the child node is a text node, extract the text content corresponding to the child node.

[0074] For example, in the DOM tree, the node mounted under the target code tag node can be an element node, a text node, a comment node, etc. Write a recursive function that receives a node as a parameter and visits all child nodes of the node, starting from the target code tag node and visiting its direct child nodes in turn.

[0075] Among them, in DOM, each node has a type attribute to indicate the type of the node, such as element node, text node, comment node, etc. Exemplarily, in JavaScript and other programming languages ​​and libraries that support DOM operations, node type attributes such as "nodeType" are used to indicate the type of the node, and each node type corresponds to a specific integer value. For example, the value of the element node (ELEMENT_NODE) ​​is 1, indicating an element in an HTML or XML document; the value of the text node (TEXT_NODE) ​​is 3, indicating the text content in an element or attribute, which is used to store actual text data. The value of the comment node (COMMENT_NODE) ​​is 8, indicating a comment in an HTML or XML document. In JavaScript, the node type attribute can be obtained by accessing the "nodeType" attribute of the node. If the node type attribute of the accessed node is a text node, the "nodeValue" attribute or the "textContent" attribute can be used to obtain the text content in the target code tag.

[0076] Optionally, the text content in the target tag and the class name corresponding to the target code tag are obtained, and the class name is used as an attribute of the text content. The class name and the text content are saved as structured data in the form of a key-value pair, stored in the database of the web content acquisition artificial intelligence system, and an application programming interface is provided for other systems to call.

[0077] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method of the present application for implementing intelligent AI to obtain web page content based on LLM. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0078] The present application provides a device for implementing intelligent AI to obtain web page content based on LLM. The device for implementing intelligent AI to obtain web page content based on LLM includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for implementing intelligent AI to obtain web page content based on LLM in the above-mentioned embodiment 1.

[0079] Reference below Figure 4 , which shows a schematic diagram of the structure of a device suitable for implementing an embodiment of the present application to implement intelligent AI acquisition of web page content based on LLM. The device in the embodiment of the present application to implement intelligent AI acquisition of web page content based on LLM may include but is not limited to mobile terminals such as laptops, digital broadcast receivers, etc., and fixed terminals such as desktop computers, etc. Figure 4 The device shown for implementing intelligent AI to obtain web page content based on LLM is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0080] like Figure 4 As shown, the device for implementing intelligent AI to obtain web page content based on LLM may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the device for implementing intelligent AI to obtain web page content based on LLM are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the device for obtaining web page content based on LLM to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a device for obtaining web page content based on LLM with intelligent AI with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0081] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0082] The device for realizing intelligent AI acquisition of webpage content based on LLM provided by the present application adopts the method for realizing intelligent AI acquisition of webpage content based on LLM in the above-mentioned embodiment, which can solve the technical problem of high time cost of crawling webpage content. Compared with the prior art, the beneficial effects of the device for realizing intelligent AI acquisition of webpage content based on LLM provided by the present application are the same as the beneficial effects of the method for realizing intelligent AI acquisition of webpage content based on LLM provided by the above-mentioned embodiment, and the other technical features in the device for realizing intelligent AI acquisition of webpage content based on LLM are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0083] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0084] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0085] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for implementing intelligent AI to obtain web page content based on LLM in the above-mentioned embodiment.

[0086] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequencies (RF, Radio Frequency), etc., or any suitable combination of the above.

[0087] The computer-readable storage medium may be included in a device for implementing intelligent AI to obtain web page content based on LLM; or may exist independently without being assembled into a device for implementing intelligent AI to obtain web page content based on LLM.

[0088] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a device that implements intelligent AI to obtain webpage content based on LLM, the device that implements intelligent AI to obtain webpage content based on LLM can be written in one or more programming languages ​​or a combination thereof to write computer program codes for performing the operations of the present application. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0090] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0091] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for implementing intelligent AI to obtain web page content based on LLM, which can solve the technical problem of high time cost of crawling web page content. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the method for implementing intelligent AI to obtain web page content based on LLM provided in the above-mentioned embodiment, and will not be repeated here.

[0092] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for implementing intelligent AI to obtain web page content based on LLM as described above.

[0093] The computer program product provided by this application can solve the technical problem of high time cost of crawling web page content. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the method for realizing intelligent AI acquisition of web page content based on LLM provided in the above embodiment, which will not be repeated here.

[0094] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application. < / h1> < / h2> < / header> < / h1>

Claims

1. A method for realizing intelligent AI acquisition of webpage content based on LLM, characterized in that: The method comprises: Receive a search instruction input by a user, input the search instruction into a trained semantic parsing model, and after the semantic parsing model converts the search instruction into a query parameter, send the query parameter to a search engine; Receiving web pages acquired by the search engine according to the query parameters, and determining a target web page from the web pages according to a preset screening rule; Based on the authorization result obtained in advance, the HTML source code of the target webpage is crawled, the HTML source code is input into the trained LLM model, and after the LLM model determines the target code tag according to the HTML source code, the text content in the target code tag is extracted.

2. The method for realizing intelligent AI acquisition of webpage content based on LLM according to claim 1, characterized in that: The step of converting the search instruction into a query parameter by the semantic parsing model comprises: The semantic parsing model receives the search instruction and performs word segmentation on the search instruction to obtain at least one word; The semantic parsing model identifies entity words in the search instruction according to a preset named entity recognition algorithm, and uses the entity words as the query parameters.

3. The method for realizing intelligent AI acquisition of webpage content based on LLM according to claim 2, characterized in that: The step of determining the target web page from the web pages according to the preset screening rules comprises: Obtaining a preset screening element, wherein the screening element includes the number of times the query parameter appears in the webpage, the update time of the webpage, and the source of the webpage; Determine the screening element score corresponding to each web page based on the preset element weight, and perform weighted summation on the screening element scores to obtain the score of each web page; The web page with a score greater than a preset threshold is used as the target web page.

4. The method for realizing intelligent AI acquisition of webpage content based on LLM according to claim 1, characterized in that: The step of determining the target code tag according to the HTML source code by the LLM model comprises: Receiving the HTML source code, parsing the HTML source code into a DOM tree, and constructing tag relationship graph data according to the DOM tree; According to the preset internal cluster compactness, hierarchical clustering analysis is performed on the tag nodes in the DOM tree to obtain tag hierarchical clustering data; According to the label relationship graph data and the label hierarchical clustering data, matching is performed in a preset knowledge base to determine the webpage type of the target webpage; Based on the webpage type of the target webpage, a target code tag in the HTML source code is determined.

5. The method for realizing intelligent AI acquisition of webpage content based on LLM according to claim 4, characterized in that: The step of determining the target code tag in the HTML source code based on the webpage type of the target webpage comprises: Determining a valid code tag corresponding to the webpage type according to the preset knowledge base; Based on the effective code tag, the target code tag is determined in the DOM tree.

6. The method for realizing intelligent AI acquisition of webpage content based on LLM according to claim 5, characterized in that: The step of determining the target code tag in the DOM tree based on the valid code tag comprises: Read the first category name corresponding to the code tag in the DOM tree, and the second category name corresponding to the valid code tag; Convert the first category name and the second category name into an embedding vector, and calculate the similarity between the first category name and the second category name; If the similarity between the first category name and the second category name is greater than a preset threshold, the code label corresponding to the first category name is determined to be the target code label.

7. The method for realizing intelligent AI acquisition of webpage content based on LLM according to claim 5, characterized in that: The step of extracting the text content in the target code tag comprises: Traverse the child nodes of the target code tag node in the DOM tree; Access the node type attribute of the child node; If the node type attribute of the child node is a text node, the text content corresponding to the child node is extracted.

8. A device for realizing intelligent AI acquisition of webpage content based on LLM, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for acquiring web page content by intelligent AI based on LLM as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the method for obtaining web page content based on intelligent AI based on LLM as described in any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the method for acquiring web page content by intelligent AI based on LLM as described in any one of claims 1 to 7 are implemented.

Images