Vertical website information extraction method, device, equipment and medium based on large model

Abstract

The present invention discloses a method, device, equipment and medium for extracting vertical website information based on a large model. According to the technical solution provided by the present invention, a large language model is used to extract the first attribute text information corresponding to the target attribute from the seed web page selected from the vertical field website; the correct node is screened from the node corresponding to the information, and the absolute path expression of the XPath of the correct node is determined; the anchor node is determined from the DOM tree based on the absolute path expression, and the XPath final expression is constructed based on the relative position of the correct node and the anchor node; the second attribute text information corresponding to the target attribute is extracted from the vertical field website using the XPath final expression. Through the present invention, the correct node and the anchor node are determined from the seed web page in the vertical field website, and the XPath final expression derived from the relative position of the two is used to extract the target information from the website, thereby achieving a lower cost and more accurate extraction of the target information without the need for model training.

Description

Technical Field

[0001] The present invention relates to the field of website information screening, and in particular to a method, device, computing equipment and computer storage medium for extracting vertical website information based on a large model. Background Art

[0002] With the development of internet technology, the web has penetrated every aspect of social life. Different verticals have their own corresponding websites, which aggregate a wealth of relevant information. For example, product details pages on e-commerce websites include information such as product name, price, and description, while sports websites contain information such as the name, height, weight, and team of athletes in different sports or teams. To obtain this information, we need to extract the corresponding fields from the HTML code of the webpage.

[0003] Currently, there are three main approaches to information extraction. The first is based on the webpage's DOM tree, using manually written XPaths to extract information. However, this approach relies heavily on manual labor, making it difficult to implement when extracting information from large quantities of websites due to the high labor and time costs. Furthermore, manually written rules struggle to adapt to dynamic changes in website content and structure. The second approach relies on deep learning methods, which serialize documents into word sequences and then use recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or attention networks to extract text fragments corresponding to structured fields. However, this approach requires a large number of labeled samples to train the neural network, resulting in high training costs and limited accuracy when training data is limited. The third approach uses prompts from a large language model to extract webpage content. However, this approach relies heavily on the performance of the large language model. Given its limitations in inference speed and runtime costs, it suffers from low efficiency and high cost when processing repetitive and similar webpage information extraction tasks in batches. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a vertical website information extraction method based on a large model and a corresponding vertical website information extraction device, computing equipment and computer storage medium based on a large model.

[0005] According to one aspect of the present invention, a method for extracting vertical website information based on a large model is provided, the method comprising:

[0006] Using a large language model, extract the first attribute text information corresponding to the target attribute from seed web pages selected from vertical field websites;

[0007] Filtering the nodes corresponding to the first attribute text information in the seed webpage to obtain a correct node corresponding to the first attribute text information, and determining an XPath absolute path expression corresponding to the correct node;

[0008] Determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and construct the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node;

[0009] The second attribute text information corresponding to the target attribute is extracted from the vertical field website using the XPath final expression corresponding to the target attribute.

[0010] In the above solution, the step of extracting the first attribute text information corresponding to the target attribute from the seed webpage using the large language model further includes:

[0011] Randomly select at least one seed webpage from vertical field websites;

[0012] Convert the HTML text information of the seed web page into text information in Mark-down format;

[0013] The large language model is used to extract first attribute text information corresponding to the target attribute from the text information of the seed webpage according to the target attribute determined by the user.

[0014] In the above solution, the step of filtering the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information and determining the absolute XPath expression corresponding to the correct node further includes:

[0015] By traversing the DOM tree corresponding to the seed webpage, identifying and screening the node to be tested containing the first attribute text information;

[0016] Filtering the nodes to be tested to obtain the correct node corresponding to the first attribute text information;

[0017] By tracing back from the correct node to the root node of the DOM tree, the absolute XPath expression corresponding to the correct node of each seed web page is determined.

[0018] In the above solution, the step of screening the nodes to be tested to obtain the correct node corresponding to the first attribute text information further includes:

[0019] Modify the first attribute text information in each node to be tested into the discrimination text information;

[0020] Extracting the modified third attribute text information of each node to be tested using the large language model, and determining whether the third attribute text information of each node to be tested is the same as the discrimination text information;

[0021] The node to be tested whose third attribute text information is identical to the discrimination text information is determined as the correct node.

[0022] In the above solution, the XPath-based absolute path expression determines the anchor node from the DOM tree corresponding to the seed webpage, and constructs the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node, further comprising:

[0023] The absolute path expression of XPath corresponding to the correct node extracts the longest common path of each correct node;

[0024] Locate the common nodes based on the longest common path of each correct node;

[0025] In the text information contained in the common node, the anchor point information is identified using the large language model, and the node where the anchor point information is located is determined as the anchor node;

[0026] Based on the relative positions of the correct node and the anchor node, the final XPath expression corresponding to the target attribute is constructed.

[0027] In the above solution, identifying anchor information from the text information contained in the common node using a large language model further includes:

[0028] Get all HTML text information contained in each public node;

[0029] Using the first attribute text information, matching is performed on all HTML text information contained in each common node to generate an input-output pair;

[0030] Construct information extraction examples based on input-output pairs;

[0031] By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

[0032] In the above solution, the construction of the final XPath expression corresponding to the target attribute based on the relative positions of the correct node and the anchor node further includes:

[0033] Determine the relative position of the correct node and the anchor node based on the position of the correct node and the position of the anchor node;

[0034] According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

[0035] According to another aspect of the present invention, a vertical website information extraction device based on a large model is provided, comprising: a first information extraction module, a node screening module, an expression generation module and a second information extraction module; wherein,

[0036] The first information extraction module is configured to extract first attribute text information corresponding to a target attribute from a seed web page selected from a vertical field website using a large language model;

[0037] The node screening module is used to screen the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information, and determine the absolute path expression of the XPath corresponding to the correct node;

[0038] The expression generation module is used to determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and to construct the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node;

[0039] The second information extraction module is used to extract the second attribute text information corresponding to the target attribute from the vertical field website using the XPath final expression corresponding to the target attribute.

[0040] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0041] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned method for extracting vertical website information based on a large model.

[0042] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned method for extracting vertical website information based on a large model.

[0043] According to the technical solution provided by the present invention, a large language model is used to extract first attribute text information corresponding to a target attribute from a seed web page selected from a vertical field website; the correct node corresponding to the first attribute text information is screened from the nodes corresponding to the first attribute text information in the seed web page, and the absolute path expression of XPath corresponding to the correct node is determined; the anchor node is determined from the DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and the final XPath expression corresponding to the target attribute is constructed based on the relative positions of the correct node and the anchor node; the final XPath expression corresponding to the target attribute is used to extract second attribute text information corresponding to the target attribute from the vertical field website. By selecting seed web pages from vertical field websites, the HTML text information of the seed web pages is formatted, and then the large language model is used to extract the text information corresponding to the target attribute determined by the user. The format conversion effectively removes redundant codes and tags, retains the key text information, and because the large language model is mainly trained on natural text, the text content is used instead of the HTML code as the processing object of the large model, which improves the processing efficiency and accuracy of the model; by screening the nodes corresponding to the target information from the DOM tree, and then comparing and checking again after modification, the correct nodes that actually need to be extracted are more accurately determined, avoiding other similar content in the web page from interfering with the information extraction, and then causing errors when extracting from other web pages, which greatly improves the accuracy of information extraction; by finding the longest common path from the absolute path of each correct node determined and further determining the common node, the text information under the node is extracted, effectively removing the web page text that is not related to the target attribute, and simplifying the extraction process. Subsequently, by pairing the text information of the seed web page with the target text information extracted previously, an example is constructed for the large language model to identify the characteristic elements that appear together with the target text information as anchor information, further determine the anchor node, and finally construct the final expression according to the relative position of the correct node and the anchor node. In this way, the target attribute information is extracted from all web pages of the vertical website. Information collection is carried out based on the anchor node, which ensures the accuracy of information extraction when the path changes and avoids the risk of collection failure. In the overall process, the complex extraction task is decomposed into multiple subtasks, which significantly reduces the dependence on the performance of the large language model and eliminates the model training process. When facing a large number of web pages of vertical websites, the large language model only needs to be called a very small number of times, which greatly saves the operating cost. The extraction work is carried out based on the anchor nodes related to the target attribute in the content or attribute value, which not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, enhances the robustness of the extraction process, and effectively expands the applicable scenarios.

[0044] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0047] Figure 1 A schematic flow chart of a method for extracting vertical website information based on a large model according to an embodiment of the present invention is shown;

[0048] Figure 2 A schematic flow chart of a method for extracting target attribute text information from a seed web page according to the present invention is shown;

[0049] Figure 3 A schematic flow chart showing a method for determining an absolute path expression of an XPath of a correct node according to an embodiment of the present invention is shown;

[0050] Figure 4 A schematic diagram of web page text information extraction and DOM tree according to an embodiment of the present invention is shown;

[0051] Figure 5 A schematic flow chart of a method for determining an XPath final expression for information extraction according to an embodiment of the present invention is shown;

[0052] Figure 6 The following is a structural block diagram of a vertical website information extraction device based on a large model according to an embodiment of the present invention;

[0053] Figure 7 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0055] Figure 1 A flowchart of a method for extracting vertical website information based on a large model according to an embodiment of the present invention is shown. The method includes the following steps:

[0056] Step S101 : using a large language model, extracting first attribute text information corresponding to a target attribute from seed web pages selected from vertical field websites.

[0057] Preferably, the target attribute is the attribute to be extracted selected according to user needs.

[0058] Step S102 : Filter the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information, and determine the absolute path expression of the XPath corresponding to the correct node.

[0059] Preferably, the method utilizes XPath language to capture elements.

[0060] Step S103 : determining the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and constructing the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node.

[0061] Step S104 : using the final XPath expression corresponding to the target attribute, extract the second attribute text information corresponding to the target attribute from the vertical field website.

[0062] According to the large model-based vertical website information extraction method provided in this embodiment, the large language model is used to extract the first attribute text information corresponding to the target attribute from the seed web page selected from the vertical field website; the correct node corresponding to the first attribute text information is screened from the node corresponding to the first attribute text information in the seed web page, and the absolute path expression of XPath corresponding to the correct node is determined; the anchor node is determined from the DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and the final XPath expression corresponding to the target attribute is constructed based on the relative position of the correct node and the anchor node; the final XPath expression corresponding to the target attribute is used to extract the second attribute text information corresponding to the target attribute from the vertical field website. Through the large model-based vertical website information extraction method provided by this embodiment, the large language model is used to select seed web pages from the vertical field website to extract text information corresponding to the target attribute determined by the user, and the nodes corresponding to the target information are screened from the DOM tree to accurately determine the correct node where information extraction is actually required, thereby avoiding interference of other similar content in the web page with the information extraction, and greatly improving the accuracy of information extraction; by determining the anchor node from the DOM tree based on the XPath absolute path expression, and finally constructing the final expression according to the relative position of the correct node and the anchor node, the target attribute information of all web pages of the vertical field website is extracted, and information is collected based on the anchor node, which ensures the accuracy of information extraction when the path changes and avoids the risk of collection failure. According to the anchor node related to the target attribute in the content or attribute value, it not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, and effectively expands the applicable scenarios.

[0063] Figure 2 A schematic flow chart of a method for extracting target attribute text information from a seed web page according to the present invention is shown;

[0064] like Figure 2 As shown, the method includes the following steps:

[0065] Step S201: randomly select at least one seed webpage from vertical field websites.

[0066] Preferably, the vertical field website can be a single website or a series of websites in the vertical field.

[0067] Preferably, the number of seed web pages may be 2-3.

[0068] Step S202: converting the HTML text information of the seed web page into text information in Mark-down format.

[0069] By format conversion, redundant JavaScript code and HTML tags are removed, retaining key text information; and because large language models are mainly trained on natural text, they are more efficient and accurate in processing text content than HTML code.

[0070] Step S203 : using the large language model, extracting first attribute text information corresponding to the target attribute from the text information of the seed web page according to the target attribute determined by the user.

[0071] Preferably, to ensure that the large language model can accurately understand the instructions and maintain the consistency of the format, an attribute example is provided for a vertical field website. Based on this example, information is extracted from the seed webpage of the vertical field website. The example prompts are as follows:

[0072]

[0073]

[0074] Taking a book website as an example, the target attributes to be extracted include the book title, author, ISBN, publisher, and publication date. Therefore, the first sentence in the example should be fully expressed as: "Please extract the book's title, author, ISBN, publisher, and publication date." The example is filled in as follows:

[0075]

[0076] Then, the examples are used as prompts and a large language model is used to extract information.

[0077] According to the above method, the text information corresponding to the target attribute determined by the user can be extracted from the seed web pages selected from the vertical field websites by using the large language model, and the correct nodes that actually need to be extracted can be accurately determined by screening the nodes corresponding to the target information from the DOM tree, thereby avoiding interference of other similar contents in the web page on the information extraction and greatly improving the accuracy of information extraction; at the same time, by setting examples of information extraction, it is easier for the large language model to understand the instructions and ensure the consistency of the format in the subsequent extraction process.

[0078] Figure 3 A schematic flow chart showing a method for determining an absolute path expression of an XPath of a correct node according to an embodiment of the present invention is shown;

[0079] like Figure 3 As shown, the method includes the following steps:

[0080] Step S301 : traverse the DOM tree corresponding to the seed webpage to identify and select the node to be tested containing the first attribute text information.

[0081] Step S302: Filter the nodes to be tested to obtain the correct nodes corresponding to the first attribute text information.

[0082] Specifically, the step of screening the nodes to be tested to obtain the correct node corresponding to the first attribute text information further includes:

[0083] Modify the first attribute text information in each node to be tested into the discrimination text information;

[0084] Extracting the modified third attribute text information of each node to be tested using the large language model, and determining whether the third attribute text information of each node to be tested is the same as the discrimination text information;

[0085] The node to be tested whose third attribute text information is identical to the discrimination text information is determined as the correct node.

[0086] Step S303 : determining the absolute XPath expression corresponding to the correct node of each seed web page by tracing back from the correct node to the root node of the DOM tree.

[0087] Preferably, starting from the root node of the DOM tree, the child node is selected and determined according to the index position in the parent node.

[0088] For example, extract the name of the team a player belongs to, such as Figure 4 As shown, Figure 4 A schematic diagram of web page text information extraction and DOM tree according to an embodiment of the present invention is shown; by screening, two places containing the first attribute text information, that is, the team name "Boston Celtics" are framed. "Boston Celtics" is modified to the discriminant text information "New Jersey Nets", and then the information is extracted again through the large language model to obtain the third attribute text information. By comparison, the team name on the left is the same as the discriminant text information, which is changed to "New Jersey Nets", while the right side does not change. Therefore, it is found that the text that truly identifies the team to which the player belongs is located in the player introduction area on the left, rather than the link area on the right, and the team name on the left is determined to be the correct node. At the same time, based on the DOM tree on the right, the correct node of the team name can be traced back to the root node of the DOM tree to determine the absolute path expression of its corresponding XPath.

[0089] According to the above method, the correct nodes that actually need to be extracted can be more accurately determined by filtering the nodes corresponding to the target information from the DOM tree and then comparing and checking them again after modification, thereby avoiding interference with information extraction caused by other similar content in the web page, which in turn leads to errors when extracting from other web pages, greatly improving the accuracy of information extraction.

[0090] Figure 5 A schematic flow chart of a method for determining an XPath final expression for information extraction according to an embodiment of the present invention is shown;

[0091] like Figure 5 As shown, the method includes the following steps:

[0092] Step S501 : extracting the longest common path of each correct node from the absolute path expression of the XPath corresponding to the correct node.

[0093] Step S502: locating a common node based on the longest common path of each correct node.

[0094] Step S503: Using the large language model, identify anchor point information in the text information included in the common node, and determine the node where the anchor point information is located as the anchor node.

[0095] Specifically, identifying anchor information from the text information contained in the common node using a large language model further includes:

[0096] Get all HTML text information contained in each public node;

[0097] Using the first attribute text information, matching is performed on all HTML text information contained in each common node to generate an input-output pair;

[0098] Construct information extraction examples based on input-output pairs;

[0099] By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

[0100] For example, based on the interactive prompt text provided to the large language model, HTML text information is extracted and matched to generate input-output pairs, where the prompt text is as follows:

[0101]

[0102] Among them, the "html_content" and "attribute_text" fields are respectively filled with the corresponding HTML text content and attribute text information extracted from the seed web page mentioned above.

[0103] Step S504: constructing a final XPath expression corresponding to the target attribute based on the relative positions of the correct node and the anchor node.

[0104] Specifically, constructing the final XPath expression corresponding to the target attribute based on the relative positions of the correct node and the anchor node further includes:

[0105] Determine the relative position of the correct node and the anchor node based on the position of the correct node and the position of the anchor node;

[0106] According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

[0107] According to the above method, the longest common path can be found from the absolute paths of the correct nodes determined, and the common node can be further determined to extract the text information under the node, effectively removing the web page text that is not related to the target attribute and simplifying the extraction process. Subsequently, by pairing the text information of the seed web page with the previously extracted target text information, an example is constructed for the large language model to identify the feature elements that appear together with the target text information as anchor information, further determining the anchor node, and finally constructing the final expression according to the relative position of the correct node and the anchor node. In this way, the target attribute information is extracted from all web pages of the vertical field website. Information collection is performed based on the anchor node, which ensures the accuracy of information extraction when the path changes and avoids the risk of collection failure. In the overall process, the complex extraction task is decomposed into multiple subtasks, which significantly reduces the dependence on the performance of the large language model. The extraction work is performed based on the anchor nodes related to the target attribute in the content or attribute value, which not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, enhances the robustness of the extraction process, and effectively expands the applicable scenarios.

[0108] Figure 6 FIG. 1 shows a structural block diagram of a vertical website information extraction device based on a large model according to an embodiment of the present invention. Figure 6 As shown, the system includes: a first information extraction module 601, a node screening module 602, an expression generation module 603 and a second information extraction module 604; wherein,

[0109] The first information extraction module 601 is configured to extract first attribute text information corresponding to a target attribute from seed web pages selected from vertical domain websites using a large language model.

[0110] Specifically, the first information extraction module 601 is further configured to:

[0111] Randomly select at least one seed webpage from vertical field websites;

[0112] Convert the HTML text information of the seed web page into text information in Mark-down format;

[0113] Using a large language model, the first attribute text information corresponding to the target attribute is extracted from the text information of the seed web page according to the target attribute determined by the user.

[0114] The node screening module 602 is configured to screen the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information, and determine the absolute XPath expression corresponding to the correct node.

[0115] Specifically, the node screening module 602 is further configured to:

[0116] By traversing the DOM tree corresponding to the seed webpage, identifying and screening the node to be tested containing the first attribute text information;

[0117] Filtering the nodes to be tested to obtain the correct node corresponding to the first attribute text information;

[0118] By tracing back from the correct node to the root node of the DOM tree, the absolute XPath expression corresponding to the correct node of each seed web page is determined.

[0119] Preferably, the node screening module 602 is further configured to:

[0120] Modify the first attribute text information in each node to be tested into the discrimination text information;

[0121] Extracting the modified third attribute text information of each node to be tested using the large language model, and determining whether the third attribute text information of each node to be tested is the same as the discrimination text information;

[0122] The node to be tested whose third attribute text information is identical to the discrimination text information is determined as the correct node.

[0123] The expression generation module 603 is used to determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and construct the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node.

[0124] Specifically, the expression generation module 603 is further used to:

[0125] The absolute path expression of XPath corresponding to the correct node extracts the longest common path of each correct node;

[0126] Locate the common nodes based on the longest common path of each correct node;

[0127] In the text information contained in the common node, the anchor point information is identified using the large language model, and the node where the anchor point information is located is determined as the anchor node;

[0128] Based on the relative positions of the correct node and the anchor node, the final XPath expression corresponding to the target attribute is constructed.

[0129] Specifically, the expression generation module 603 is further used to:

[0130] Get all HTML text information contained in each public node;

[0131] Using the first attribute text information, matching is performed on all HTML text information contained in each common node to generate an input-output pair;

[0132] Construct information extraction examples based on input-output pairs;

[0133] By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

[0134] Preferably, the expression generation module 603 is further used to:

[0135] Determine the relative position of the correct node and the anchor node based on the position of the correct node and the position of the anchor node;

[0136] According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

[0137] The second information extraction module 604 is configured to extract second attribute text information corresponding to the target attribute from the vertical field website using the XPath final expression corresponding to the target attribute.

[0138] According to the large model-based vertical website information extraction device provided in this embodiment, it includes: a first information extraction module, a node screening module, an expression generation module and a second information extraction module; wherein, the first information extraction module is used to use the large language model to extract the first attribute text information corresponding to the target attribute from the seed web page selected from the vertical field website; the node screening module is used to filter the correct node corresponding to the first attribute text information from the node corresponding to the first attribute text information in the seed web page, and determine the absolute path expression of the XPath corresponding to the correct node; the expression generation module is used to determine the anchor node from the DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and construct the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node; the second information extraction module is used to use the XPath final expression corresponding to the target attribute to extract the second attribute text information corresponding to the target attribute from the vertical field website.The vertical website information extraction device based on the big model provided by this embodiment selects seed web pages from vertical field websites, converts the format of the HTML text information of the seed web pages, and then uses the big language model to extract the text information corresponding to the target attribute determined by the user. The format conversion effectively removes redundant codes and tags, retains key text information, and because the big language model is mainly trained on natural text, the text content is used instead of the HTML code as the processing object of the big model, which improves the processing efficiency and accuracy of the model; by screening the nodes corresponding to the target information from the DOM tree, and then comparing and checking again after modification, the correct nodes that actually need to be extracted are more accurately determined, avoiding interference with information extraction caused by other similar content in the web page, which leads to errors when extracting from other web pages, and greatly improving the accuracy of information extraction; by finding the longest common path from the absolute paths of the determined correct nodes and further determining the common node, the text information under the node is extracted, and the information irrelevant to the target attribute is effectively removed. The extraction process is simplified by pairing the text information of the seed web page with the target text information extracted previously. Then, examples are constructed for the large language model to identify the characteristic elements that appear together with the target text information as anchor information, and the anchor nodes are further determined. Finally, the final expression is constructed according to the relative position of the correct node and the anchor node. In this way, the target attribute information is extracted from all web pages of the vertical website. Information collection is carried out based on the anchor nodes, which ensures the accuracy of information extraction when the path changes and avoids the risk of collection failure. In the overall process, the complex extraction task is decomposed into multiple subtasks, which significantly reduces the dependence on the performance of the large language model and eliminates the model training process. When dealing with a large number of web pages of vertical websites, the large language model only needs to be called a very small number of times, which greatly saves the operating cost. The extraction work is carried out based on the anchor nodes related to the target attribute in the content or attribute value, which not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, enhances the robustness of the extraction process, and effectively expands the applicable scenarios.

[0139] The present invention also provides a non-volatile computer storage medium, which stores at least one executable instruction. The executable instruction can execute the large model-based vertical website information extraction method in any of the above method embodiments.

[0140] Figure 7 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0141] like Figure 7As shown, the computing device may include: a processor (processor) 702 , a communications interface (Communications Interface) 704 , a memory (memory) 706 , and a communication bus 708 .

[0142] in:

[0143] The processor 702 , the communication interface 704 , and the memory 706 communicate with each other via a communication bus 708 .

[0144] The communication interface 704 is used to communicate with other devices such as clients or other servers.

[0145] The processor 702 is used to execute the program 710, and specifically can execute the relevant steps in the above-mentioned embodiment of the vertical website information extraction method based on the large model.

[0146] Specifically, the program 710 may include program codes, which include computer operating instructions.

[0147] Processor 702 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a computing device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0148] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0149] Program 710 can be specifically used to enable processor 702 to execute a vertical website information extraction method based on a large model in any of the above-mentioned method embodiments. The specific implementation of each step in program 710 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned embodiment of a vertical website information extraction method based on a large model, and will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described equipment and modules can refer to the corresponding process description in the aforementioned method embodiment, and will not be repeated here.

[0150] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0151] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0152] Similarly, it should be understood that in order to streamline the present disclosure and aid understanding of one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0153] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0154] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0155] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in accordance with the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a portion or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0156] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for extracting vertical website information based on a large model, comprising: Using a large language model, extract the first attribute text information corresponding to the target attribute from seed web pages selected from vertical field websites; The correct node corresponding to the first attribute text information is screened from the nodes corresponding to the first attribute text information in the seed webpage, and the absolute path expression of the XPath corresponding to the correct node is determined; wherein, By traversing the DOM tree corresponding to the seed webpage, identifying and screening the node to be tested containing the first attribute text information; Filtering the correct nodes corresponding to the first attribute text information from the nodes to be tested; wherein the first attribute text information in each node to be tested is modified into discriminant text information; using the large language model to extract the modified third attribute text information of each node to be tested, and determining whether the third attribute text information of each node to be tested is the same as the discriminant text information; determining the node to be tested whose third attribute text information is the same as the discriminant text information as the correct node; By tracing back from the correct node to the root node of the DOM tree, the absolute XPath path expression corresponding to the correct node of each seed web page is determined; Determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and construct the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node; The second attribute text information corresponding to the target attribute is extracted from the vertical field website using the XPath final expression corresponding to the target attribute.

2. The method according to claim 1, characterized in that The extracting first attribute text information corresponding to the target attribute from the seed webpage using the large language model further includes: Randomly select at least one seed webpage from vertical field websites; Convert the HTML text information of the seed web page into text information in Mark-down format; The large language model is used to extract first attribute text information corresponding to the target attribute from the text information of the seed webpage according to the target attribute determined by the user.

3. The method according to claim 1, characterized in that The XPath-based absolute path expression determines the anchor node from the DOM tree corresponding to the seed webpage, and constructs the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node, further comprising: The absolute path expression of XPath corresponding to the correct node extracts the longest common path of each correct node; Locate the common nodes based on the longest common path of each correct node; In the text information contained in the common node, the anchor point information is identified using the large language model, and the node where the anchor point information is located is determined as the anchor node; Based on the relative positions of the correct node and the anchor node, the final XPath expression corresponding to the target attribute is constructed.

4. The method according to claim 3, characterized in that The method of identifying anchor point information from the text information contained in the common node using a large language model further includes: Get all HTML text information contained in each public node; Using the first attribute text information, matching is performed on all HTML text information contained in each common node to generate an input-output pair; Construct information extraction examples based on input-output pairs; By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

5. The method according to claim 3, characterized in that The step of constructing a final XPath expression corresponding to the target attribute based on the relative positions of the correct node and the anchor node further includes: Determine the relative position of the correct node and the anchor node based on the position of the correct node and the position of the anchor node; According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

6. A vertical website information extraction device based on a large model, comprising: A first information extraction module, a node screening module, an expression generation module and a second information extraction module; wherein, The first information extraction module is configured to extract first attribute text information corresponding to a target attribute from a seed web page selected from a vertical field website using a large language model; The node screening module is used to screen the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information, and determine the absolute path expression of the XPath corresponding to the correct node; The expression generation module is used to determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and to construct the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node; wherein, By traversing the DOM tree corresponding to the seed webpage, identifying and screening the node to be tested containing the first attribute text information; Filtering the correct nodes corresponding to the first attribute text information from the nodes to be tested; wherein the first attribute text information in each node to be tested is modified into discriminant text information; using the large language model to extract the modified third attribute text information of each node to be tested, and determining whether the third attribute text information of each node to be tested is the same as the discriminant text information; determining the node to be tested whose third attribute text information is the same as the discriminant text information as the correct node; By tracing back from the correct node to the root node of the DOM tree, the absolute XPath path expression corresponding to the correct node of each seed web page is determined; The second information extraction module is used to extract the second attribute text information corresponding to the target attribute from the vertical field website using the XPath final expression corresponding to the target attribute.

7. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to a vertical website information extraction method based on a large model as described in any one of claims 1-5.

8. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to a vertical website information extraction method based on a large model as described in any one of claims 1-5.

Images