Vertical website information extraction method and device based on large model, equipment and medium

Abstract

The invention discloses a vertical website information extraction method and device based on a large model, equipment and a medium. According to the technical scheme provided by the invention, the method comprises the following steps: extracting first attribute text information corresponding to a target attribute from a seed webpage selected from a vertical domain website by utilizing a large language model; screening correct nodes from the nodes corresponding to the information, and determining an absolute path expression of XPath of the correct nodes; determining anchor nodes from the DOM tree based on the absolute path expression, and constructing an XPath final expression based on the relative positions of the correct nodes and the anchor nodes; and extracting second attribute text information corresponding to the target attribute from the vertical domain website by utilizing the XPath final expression. According to the method, the correct node and the anchor node are determined through the seed webpage in the vertical domain website, and the target information is extracted from the website through the XPath final expression obtained through the relative positions of the correct node and the anchor node, so that the target information is extracted more accurately with lower cost under the condition that model training is not needed.

Description

Technical Field

[0001] The present invention relates to the field of website information screening, and in particular to a method, device, computing equipment and computer storage medium for extracting vertical website information based on a large model. Background Art

[0002] With the development of Internet technology, the network has penetrated into all areas of social life. For different vertical fields, there are corresponding vertical field websites, which collect a lot of relevant information in the field. For example, the product details page of the e-commerce website contains the name, price, introduction and other information of the product, and the sports website contains the name, height, weight, team and other information of athletes in different sports or different teams. If you need to obtain this information, you need to extract the corresponding fields of the information we need from the HTML code of the web page.

[0003] In this regard, there are currently three main ways of information extraction: the first is based on the DOM tree of the web page, and information extraction is achieved by manually writing XPath. However, this method relies too much on manual labor. When extracting information from a large number of websites, it is difficult to implement due to the high manpower and time costs. When the content and structure of the website change, the manually written rules are also difficult to adapt to such dynamic changes. The second is based on deep learning methods, by serializing the document into a word sequence, and then using a recurrent neural network (RNN), a long short-term memory network (LSTM) or an attention network to extract text fragments corresponding to the structured field. However, this method requires a large number of labeled samples to complete the training of the neural network, which has a high training cost and cannot guarantee accuracy when the amount of training data is small. The third method is to use the prompts of a large language model to extract web page content, but this method relies too much on the performance of the large language model, and given the large language model's inference speed and running cost, it has low processing efficiency and high cost when batch processing repeated and similar web page information extraction tasks. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a vertical website information extraction method based on a large model and a corresponding vertical website information extraction device, computing equipment and computer storage medium based on a large model.

[0005] According to one aspect of the present invention, a method for extracting vertical website information based on a large model is provided, the method comprising:

[0006] Using a large language model, extract the first attribute text information corresponding to the target attribute from the seed web pages selected from the vertical field websites;

[0007] Filtering the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information, and determining the absolute path expression of the XPath corresponding to the correct node;

[0008] Determine the anchor node from the DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and construct the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node;

[0009] The XPath final expression corresponding to the target attribute is used to extract the second attribute text information corresponding to the target attribute from the vertical field website.

[0010] In the above solution, the extracting the first attribute text information corresponding to the target attribute from the seed web page using the large language model further includes:

[0011] Randomly select at least one seed webpage from vertical field websites;

[0012] Convert the HTML text information of the seed web page into text information in Mark-down format;

[0013] The large language model is used to extract first attribute text information corresponding to the target attribute from the text information of the seed web page according to the target attribute determined by the user.

[0014] In the above scheme, the step of screening the correct node corresponding to the first attribute text information from the nodes corresponding to the first attribute text information in the seed webpage and determining the absolute path expression of the XPath corresponding to the correct node further includes:

[0015] By traversing the DOM tree corresponding to the seed webpage, identifying and selecting the node to be tested containing the first attribute text information;

[0016] Filtering the nodes to be tested to obtain the correct nodes corresponding to the first attribute text information;

[0017] By tracing back from the correct node to the root node of the DOM tree, the absolute path expression of the XPath corresponding to the correct node of each seed web page is determined.

[0018] In the above solution, the step of screening out the correct node corresponding to the first attribute text information from the nodes to be tested further includes:

[0019] Modify the first attribute text information in each node to be tested into the discrimination text information;

[0020] Extracting the modified third attribute text information of each node to be tested by using the large language model, and determining whether the third attribute text information of each node to be tested is the same as the determination text information;

[0021] The node to be tested whose third attribute text information is the same as the discrimination text information is determined as the correct node.

[0022] In the above scheme, the XPath-based absolute path expression determines the anchor node from the DOM tree corresponding to the seed webpage, and constructs the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node, further comprising:

[0023] The absolute path expression of XPath corresponding to the correct node extracts the longest common path of each correct node;

[0024] Locate the common nodes based on the longest common path of each correct node;

[0025] In the text information contained in the common node, the anchor point information is identified by using the large language model, and the node where the anchor point information is located is determined as the anchor node;

[0026] Based on the relative positions of the correct node and the anchor node, the final XPath expression corresponding to the target attribute is constructed.

[0027] In the above scheme, the method of identifying anchor information in the text information contained in the common node using a large language model further includes:

[0028] Get all HTML text information contained in each public node;

[0029] Using the first attribute text information, matching is performed in all HTML text information contained in each public node to generate an input-output pair;

[0030] Construct information extraction examples based on input-output pairs;

[0031] By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

[0032] In the above solution, the construction of the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node further includes:

[0033] According to the position of the correct node and the position of the anchor node, determine the relative position of the correct node and the anchor node;

[0034] According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

[0035] According to another aspect of the present invention, a vertical website information extraction device based on a large model is provided, comprising: a first information extraction module, a node screening module, an expression generation module and a second information extraction module; wherein,

[0036] The first information extraction module is used to extract first attribute text information corresponding to the target attribute from the seed web pages selected from the vertical field website using the large language model;

[0037] The node screening module is used to screen the correct node corresponding to the first attribute text information from the nodes corresponding to the first attribute text information in the seed webpage, and determine the absolute path expression of the XPath corresponding to the correct node;

[0038] The expression generation module is used to determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and to construct the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node;

[0039] The second information extraction module is used to extract the second attribute text information corresponding to the target attribute from the vertical field website by using the XPath final expression corresponding to the target attribute.

[0040] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0041] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned method for extracting vertical website information based on a large model.

[0042] According to another aspect of the present invention, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned method for extracting vertical website information based on a large model.

[0043] According to the technical solution provided by the present invention, a large language model is used to extract first attribute text information corresponding to a target attribute from a seed web page selected from a vertical field website; a correct node corresponding to the first attribute text information is screened from the nodes corresponding to the first attribute text information in the seed web page, and an absolute path expression of XPath corresponding to the correct node is determined; an anchor node is determined from a DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and an XPath final expression corresponding to the target attribute is constructed based on the relative positions of the correct node and the anchor node; and a second attribute text information corresponding to the target attribute is extracted from the vertical field website using the XPath final expression corresponding to the target attribute. By selecting seed web pages from vertical field websites, the HTML text information of the seed web pages is formatted, and then the large language model is used to extract the text information corresponding to the target attribute determined by the user. The format conversion effectively removes redundant codes and tags, retains key text information, and because the large language model is mainly trained on natural text, the text content is used instead of HTML code as the processing object of the large model, which improves the processing efficiency and accuracy of the model; by screening the nodes corresponding to the target information from the DOM tree, and then comparing and checking again after modification, the correct nodes that actually need to be extracted are more accurately determined, avoiding other similar content in the web page from interfering with the information extraction, and then causing errors when extracting from other web pages, which greatly improves the accuracy of information extraction; by finding the longest common path from the absolute path of each correct node determined and further determining the common node, the text information under the node is extracted, effectively removing the web page text that is not related to the target attribute, and simplifying the extraction process Subsequently, by pairing the text information of the seed web page with the previously extracted target text information, an example is constructed so that the large language model can identify the characteristic elements that co-appear with the target text information as anchor information, further determine the anchor node, and finally construct the final expression according to the relative position of the correct node and the anchor node, so as to extract the information of the target attribute from all the web pages of the vertical website. Information collection based on the anchor node ensures the accuracy of information extraction when the path changes and avoids the risk of collection failure. In the overall process, the complex extraction task is decomposed into multiple subtasks, which significantly reduces the dependence on the performance of the large language model and eliminates the model training process. When facing a large number of web pages of vertical websites, the large language model only needs to be called very few times, which greatly saves the operating cost. The extraction work is performed based on the anchor nodes related to the target attribute in the content or attribute value, which not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, enhances the robustness of the extraction process, and effectively expands the applicable scenarios.

[0044] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0047] Figure 1 A schematic flow chart of a method for extracting vertical website information based on a large model according to an embodiment of the present invention is shown;

[0048] Figure 2 A schematic diagram showing a flow chart of a method for extracting target attribute text information from a seed web page according to the present invention;

[0049] Figure 3 A schematic diagram showing a flow chart of a method for determining an absolute path expression of an XPath of a correct node according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of web page text information extraction and DOM tree according to an embodiment of the present invention is shown;

[0051] Figure 5 A schematic flow chart of a method for determining an XPath final expression for information extraction according to an embodiment of the present invention is shown;

[0052] Figure 6 A structural block diagram of a vertical website information extraction device based on a large model according to an embodiment of the present invention is shown;

[0053] Figure 7 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0055] Figure 1 A schematic flow chart of a method for extracting vertical website information based on a large model according to an embodiment of the present invention is shown. The method comprises the following steps:

[0056] Step S101, using a large language model, extracting first attribute text information corresponding to a target attribute from seed web pages selected from vertical field websites.

[0057] Preferably, the target attribute is the attribute to be extracted selected according to user needs.

[0058] Step S102: Filter the nodes corresponding to the first attribute text information in the seed web page to obtain the correct node corresponding to the first attribute text information, and determine the absolute path expression of the XPath corresponding to the correct node.

[0059] Preferably, the method utilizes XPath language to capture elements.

[0060] Step S103, determining the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and constructing the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node.

[0061] Step S104, using the XPath final expression corresponding to the target attribute, extract the second attribute text information corresponding to the target attribute from the vertical field website.

[0062] According to the large model-based vertical website information extraction method provided in this embodiment, the large language model is used to extract the first attribute text information corresponding to the target attribute from the seed web page selected from the vertical field website; the correct node corresponding to the first attribute text information is screened from the node corresponding to the first attribute text information in the seed web page, and the absolute path expression of XPath corresponding to the correct node is determined; the anchor node is determined from the DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and the XPath final expression corresponding to the target attribute is constructed based on the relative position of the correct node and the anchor node; the second attribute text information corresponding to the target attribute is extracted from the vertical field website using the XPath final expression corresponding to the target attribute. Through the large model-based vertical website information extraction method provided by this embodiment, the large language model is used to select seed web pages from the vertical field website to extract text information corresponding to the target attribute determined by the user, and the nodes corresponding to the target information are screened from the DOM tree to accurately determine the correct nodes that actually need to be extracted for information, thereby avoiding interference with information extraction by other similar contents in the web page, and greatly improving the accuracy of information extraction; by determining the anchor node from the DOM tree based on the XPath absolute path expression, and finally constructing the final expression according to the relative position of the correct node and the anchor node, the target attribute information is extracted from all web pages of the vertical field website, and information is collected based on the anchor nodes, which ensures the accuracy of information extraction when the path changes, avoids the risk of collection failure, and is based on the anchor nodes related to the target attribute in the content or attribute value, which not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, and effectively expands the applicable scenarios.

[0063] Figure 2 A schematic diagram showing a flow chart of a method for extracting target attribute text information from a seed web page according to the present invention;

[0064] like Figure 2 As shown, the method comprises the following steps:

[0065] Step S201, randomly selecting at least one seed web page from vertical field websites.

[0066] Preferably, the vertical field website may be a single website or a series of websites in the vertical field.

[0067] Preferably, the number of seed web pages may be 2-3.

[0068] Step S202: converting the HTML text information of the seed web page into text information in Mark-down format.

[0069] By converting the format, redundant JavaScript code and HTML tags are removed, retaining key text information; and because large language models are mainly trained on natural text, they are more efficient and accurate in processing text content than HTML code.

[0070] Step S203: using the large language model, extracting first attribute text information corresponding to the target attribute from the text information of the seed web page according to the target attribute determined by the user.

[0071] Preferably, in order to ensure that the large language model can accurately understand the instructions and maintain the consistency of the format, an attribute example is provided for a vertical field website. Based on this example, information is extracted from the seed webpage of the vertical field website. The example prompts are as follows:

[0072]

[0073]

[0074] Taking the book website as an example, the target attributes to be extracted include the book title, author, ISBN, publisher, and publication date. Therefore, the first sentence in the example should be fully expressed as: "Please extract the book title, author, ISBN, publisher, and publication date." The example is filled in as follows:

[0075]

[0076] Then, the examples are used as prompts to extract information using a large language model.

[0077] According to the above method, the text information corresponding to the target attribute determined by the user can be extracted from the seed web pages selected from the vertical field websites by using the large language model, and the correct nodes actually required for information extraction can be accurately determined by screening the nodes corresponding to the target information from the DOM tree, thereby avoiding interference of other similar contents in the web page on the information extraction and greatly improving the accuracy of information extraction; at the same time, by setting examples of information extraction, it is convenient for the large language model to understand the instructions and ensures the consistency of the format in the subsequent extraction process.

[0078] Figure 3 A schematic diagram showing a flow chart of a method for determining an absolute path expression of an XPath of a correct node according to an embodiment of the present invention;

[0079] like Figure 3 As shown, the method comprises the following steps:

[0080] Step S301 , traversing the DOM tree corresponding to the seed webpage, identifying and selecting the node to be tested containing the first attribute text information.

[0081] Step S302: Filter and obtain the correct node corresponding to the first attribute text information from the nodes to be tested.

[0082] Specifically, the step of screening out the correct node corresponding to the first attribute text information from the nodes to be tested further includes:

[0083] Modify the first attribute text information in each node to be tested into the discrimination text information;

[0084] Extracting the modified third attribute text information of each node to be tested by using the large language model, and determining whether the third attribute text information of each node to be tested is the same as the determination text information;

[0085] The node to be tested whose third attribute text information is the same as the discrimination text information is determined as the correct node.

[0086] Step S303 , by tracing upward from the correct node to the root node of the DOM tree, the absolute path expression of the XPath corresponding to the correct node of each seed web page is determined.

[0087] Preferably, starting from the root node of the DOM tree, the child node is selected and determined according to the index position in the parent node.

[0088] For example, extract the name of the team a player belongs to, such as Figure 4 As shown, Figure 4 A schematic diagram of web page text information extraction and DOM tree according to an embodiment of the present invention is shown; by screening, two places containing the first attribute text information, that is, the team name "Boston Celtics" are framed. "Boston Celtics" is modified to the discriminant text information "New Jersey Nets", and then the information is extracted again through the large language model to obtain the third attribute text information. By comparison, the team name on the left is the same as the discriminant text information, which is changed to "New Jersey Nets", while the right side does not change. Therefore, it will be found that the text that truly identifies the team to which the player belongs is located in the player introduction area on the left, rather than the link area on the right, and then the team name on the left is determined to be the correct node. At the same time, based on the DOM tree on the right, the correct node of the team name can be traced back to the root node of the DOM tree to determine the absolute path expression of the corresponding XPath.

[0089] According to the above method, the correct nodes that actually need to be extracted for information can be more accurately determined by filtering the nodes corresponding to the target information from the DOM tree and comparing and checking them again after modification, thereby avoiding interference with information extraction caused by other similar contents in the web page, which in turn leads to errors when extracting from other web pages, greatly improving the accuracy of information extraction.

[0090] Figure 5 A schematic flow chart of a method for determining an XPath final expression for information extraction according to an embodiment of the present invention is shown;

[0091] like Figure 5 As shown, the method comprises the following steps:

[0092] Step S501: The absolute path expression of XPath corresponding to the correct node extracts the longest common path of each correct node.

[0093] Step S502: locate the common nodes among the correct nodes based on the longest common path among the correct nodes.

[0094] Step S503: In the text information included in the common node, the anchor point information is identified using the large language model, and the node where the anchor point information is located is determined as the anchor node.

[0095] Specifically, the method of identifying anchor information from the text information contained in the common node using a large language model further includes:

[0096] Get all HTML text information contained in each public node;

[0097] Using the first attribute text information, matching is performed in all HTML text information contained in each public node to generate an input-output pair;

[0098] Construct information extraction examples based on input-output pairs;

[0099] By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

[0100] For example, based on the interactive prompt text provided to the large language model, HTML text information is extracted and matched to generate input-output pairs, where the prompt text is as follows:

[0101]

[0102] Among them, the "html_content" and "attribute_text" fields are respectively filled with the corresponding HTML text content and attribute text information extracted from the seed web page mentioned above.

[0103] Step S504: construct a final XPath expression corresponding to the target attribute based on the relative positions of the correct node and the anchor node.

[0104] Specifically, the constructing of the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node further includes:

[0105] According to the position of the correct node and the position of the anchor node, determine the relative position of the correct node and the anchor node;

[0106] According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

[0107] According to the above method, the longest common path can be found from the absolute paths of the determined correct nodes and the common nodes can be further determined to extract the text information under the node, effectively removing the web page text that is not related to the target attribute and simplifying the extraction process. Subsequently, by pairing the text information of the seed web page with the target text information extracted previously, an example is constructed so that the large language model can identify the feature elements that co-appear with the target text information as anchor information, further determine the anchor node, and finally construct the final expression according to the relative position of the correct node and the anchor node, so as to extract the information of the target attribute from all web pages of the vertical website, and collect information based on the anchor nodes, so as to ensure the accuracy of information extraction when the path changes and avoid the risk of collection failure. In the overall process, the complex extraction task is decomposed into multiple subtasks, which significantly reduces the dependence on the performance of the large language model, and the extraction work is performed based on the anchor nodes related to the target attribute in the content or attribute value, which not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, enhances the robustness of the extraction process, and effectively expands the applicable scenarios.

[0108] Figure 6 FIG. 4 shows a structural block diagram of a vertical website information extraction device based on a large model according to an embodiment of the present invention. Figure 6 As shown, the system includes: a first information extraction module 601, a node screening module 602, an expression generation module 603 and a second information extraction module 604; wherein,

[0109] The first information extraction module 601 is used to extract first attribute text information corresponding to a target attribute from seed web pages selected from vertical domain websites using a large language model.

[0110] Specifically, the first information extraction module 601 is further used to:

[0111] Randomly select at least one seed webpage from vertical field websites;

[0112] Convert the HTML text information of the seed web page into text information in Mark-down format;

[0113] Using the large language model, the first attribute text information corresponding to the target attribute is extracted from the text information of the seed web page according to the target attribute determined by the user.

[0114] The node screening module 602 is used to screen the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information, and determine the absolute path expression of the XPath corresponding to the correct node.

[0115] Specifically, the node screening module 602 is further used to:

[0116] By traversing the DOM tree corresponding to the seed webpage, identifying and selecting the node to be tested containing the first attribute text information;

[0117] Filtering the nodes to be tested to obtain the correct nodes corresponding to the first attribute text information;

[0118] By tracing back from the correct node to the root node of the DOM tree, the absolute path expression of the XPath corresponding to the correct node of each seed web page is determined.

[0119] Preferably, the node screening module 602 is further used to:

[0120] Modify the first attribute text information in each node to be tested into the discrimination text information;

[0121] Extracting the modified third attribute text information of each node to be tested by using the large language model, and determining whether the third attribute text information of each node to be tested is the same as the determination text information;

[0122] The node to be tested whose third attribute text information is the same as the discrimination text information is determined as the correct node.

[0123] The expression generation module 603 is used to determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and to construct the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node.

[0124] Specifically, the expression generation module 603 is further used to:

[0125] The absolute path expression of XPath corresponding to the correct node extracts the longest common path of each correct node;

[0126] Locate the common nodes based on the longest common path of each correct node;

[0127] In the text information contained in the common node, the anchor point information is identified by using the large language model, and the node where the anchor point information is located is determined as the anchor node;

[0128] Based on the relative positions of the correct node and the anchor node, the final XPath expression corresponding to the target attribute is constructed.

[0129] Specifically, the expression generation module 603 is further used to:

[0130] Get all HTML text information contained in each public node;

[0131] Using the first attribute text information, matching is performed in all HTML text information contained in each public node to generate an input-output pair;

[0132] Construct information extraction examples based on input-output pairs;

[0133] By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

[0134] Preferably, the expression generation module 603 is further used to:

[0135] According to the position of the correct node and the position of the anchor node, determine the relative position of the correct node and the anchor node;

[0136] According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

[0137] The second information extraction module 604 is used to extract the second attribute text information corresponding to the target attribute from the vertical field website using the XPath final expression corresponding to the target attribute.

[0138] The vertical website information extraction device based on the big model provided in this embodiment includes: a first information extraction module, a node screening module, an expression generation module and a second information extraction module; wherein the first information extraction module is used to use the big language model to extract the first attribute text information corresponding to the target attribute from the seed web page selected from the vertical field website; the node screening module is used to filter the correct node corresponding to the first attribute text information from the node corresponding to the first attribute text information in the seed web page, and determine the absolute path expression of XPath corresponding to the correct node; the expression generation module is used to determine the anchor node from the DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and construct the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node; the second information extraction module is used to use the XPath final expression corresponding to the target attribute to extract the second attribute text information corresponding to the target attribute from the vertical field website.The vertical website information extraction device based on the big model provided by this embodiment selects seed web pages from vertical field websites, converts the format of the HTML text information of the seed web pages, and then uses the big language model to extract the text information corresponding to the target attribute determined by the user. The format conversion effectively removes redundant codes and tags, retains key text information, and because the big language model is mainly trained on natural text, the text content is used instead of the HTML code as the processing object of the big model, which improves the processing efficiency and accuracy of the model; by screening the nodes corresponding to the target information from the DOM tree, and by comparing and checking again after modification, the correct nodes that actually need to be extracted for information are more accurately determined, avoiding interference with information extraction caused by other similar content in the web page, thereby causing errors when extracting from other web pages, and greatly improving the accuracy of information extraction; by finding the longest common path from the absolute paths of the determined correct nodes and further determining the common node, the text information under the node is extracted, and the information that is not related to the target attribute is effectively removed. The extraction process is simplified by pairing the text information of the seed web page with the target text information extracted previously, and constructing examples for the large language model to identify the feature elements that co-occur with the target text information as anchor information, further determine the anchor nodes, and finally construct the final expression according to the relative positions of the correct nodes and the anchor nodes. In this way, the target attribute information is extracted from all web pages of vertical websites. Information collection is carried out based on anchor nodes, which ensures the accuracy of information extraction when the path changes and avoids the risk of collection failure. In the overall process, the complex extraction task is decomposed into multiple subtasks, which significantly reduces the dependence on the performance of the large language model and eliminates the model training process. When facing a large number of web pages of vertical websites, only a few calls to the large language model are required, which greatly saves the operating cost. The extraction work is carried out based on the anchor nodes related to the target attribute in the content or attribute value, which not only ensures the accuracy of information extraction, but also greatly improves the flexibility and generalization of information extraction, enhances the robustness of the extraction process, and effectively expands the applicable scenarios.

[0139] The present invention also provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can execute the large model-based vertical website information extraction method in any of the above method embodiments.

[0140] Figure 7 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0141] like Figure 7As shown, the computing device may include: a processor (processor) 702 , a communications interface (Communications Interface) 704 , a memory (memory) 706 , and a communication bus 708 .

[0142] in:

[0143] The processor 702 , the communication interface 704 , and the memory 706 communicate with each other via a communication bus 708 .

[0144] The communication interface 704 is used to communicate with other devices such as clients or other servers.

[0145] Processor 702 is used to execute program 710, and specifically can execute the relevant steps in the above-mentioned embodiment of the vertical website information extraction method based on a large model.

[0146] Specifically, the program 710 may include program code, and the program code includes computer operation instructions.

[0147] The processor 702 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0148] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0149] Program 710 can be specifically used to enable processor 702 to execute a vertical website information extraction method based on a large model in any of the above-mentioned method embodiments. The specific implementation of each step in program 710 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned vertical website information extraction method embodiment based on a large model, which will not be repeated here. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described equipment and modules can refer to the corresponding process description in the aforementioned method embodiment, which will not be repeated here.

[0150] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0151] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0152] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0153] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0154] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0155] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for extracting vertical website information based on a large model, comprising: Using a large language model, extract the first attribute text information corresponding to the target attribute from the seed web pages selected from the vertical field websites; Filtering the nodes corresponding to the first attribute text information in the seed webpage to obtain the correct node corresponding to the first attribute text information, and determining the absolute path expression of the XPath corresponding to the correct node; Determine the anchor node from the DOM tree corresponding to the seed web page based on the absolute path expression of XPath, and construct the final XPath expression corresponding to the target attribute based on the relative position of the correct node and the anchor node; The XPath final expression corresponding to the target attribute is used to extract the second attribute text information corresponding to the target attribute from the vertical field website.

2. The method according to claim 1, characterized in that The extracting the first attribute text information corresponding to the target attribute from the seed web page using the large language model further includes: Randomly select at least one seed webpage from vertical field websites; Convert the HTML text information of the seed web page into text information in Mark-down format; The large language model is used to extract first attribute text information corresponding to the target attribute from the text information of the seed web page according to the target attribute determined by the user.

3. The method according to claim 1, characterized in that The step of screening out a correct node corresponding to the first attribute text information from the nodes corresponding to the first attribute text information in the seed webpage, and determining an absolute path expression of XPath corresponding to the correct node, further includes: By traversing the DOM tree corresponding to the seed webpage, identifying and selecting the node to be tested containing the first attribute text information; Filtering the nodes to be tested to obtain the correct nodes corresponding to the first attribute text information; By tracing back from the correct node to the root node of the DOM tree, the absolute path expression of the XPath corresponding to the correct node of each seed web page is determined.

4. The method according to claim 3, characterized in that The step of screening out the correct node corresponding to the first attribute text information from the node to be tested further includes: Modify the first attribute text information in each node to be tested into the discrimination text information; Extracting the modified third attribute text information of each node to be tested by using the large language model, and determining whether the third attribute text information of each node to be tested is the same as the determination text information; The node to be tested whose third attribute text information is the same as the discrimination text information is determined as the correct node.

5. The method according to claim 1, characterized in that The XPath-based absolute path expression determines the anchor node from the DOM tree corresponding to the seed webpage, and constructs the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node, further comprising: The absolute path expression of XPath corresponding to the correct node extracts the longest common path of each correct node; Locate the common nodes based on the longest common path of each correct node; In the text information contained in the common node, the anchor point information is identified by using the large language model, and the node where the anchor point information is located is determined as the anchor node; Based on the relative positions of the correct node and the anchor node, the final XPath expression corresponding to the target attribute is constructed.

6. The method according to claim 5, characterized in that The method of identifying anchor point information from the text information contained in the common node using a large language model further includes: Get all HTML text information contained in each public node; Using the first attribute text information, matching is performed in all HTML text information contained in each public node to generate an input-output pair; Construct information extraction examples based on input-output pairs; By using the large language model and extracting examples, anchor information corresponding to the first attribute text information is determined from all HTML text information contained in each common node.

7. The method according to claim 5, characterized in that The step of constructing the XPath final expression corresponding to the target attribute based on the relative positions of the correct node and the anchor node further includes: According to the position of the correct node and the position of the anchor node, determine the relative position of the correct node and the anchor node; According to the relative positions of the correct node and the anchor node, an XPath expression of the relative path between the correct node and the anchor node is determined and used as the final XPath expression.

8. A vertical website information extraction device based on a large model, comprising: A first information extraction module, a node screening module, an expression generation module and a second information extraction module; wherein, The first information extraction module is used to extract first attribute text information corresponding to the target attribute from the seed web pages selected from the vertical field website using the large language model; The node screening module is used to screen the correct node corresponding to the first attribute text information from the nodes corresponding to the first attribute text information in the seed webpage, and determine the absolute path expression of the XPath corresponding to the correct node; The expression generation module is used to determine the anchor node from the DOM tree corresponding to the seed webpage based on the absolute path expression of XPath, and to construct the XPath final expression corresponding to the target attribute based on the relative position of the correct node and the anchor node; The second information extraction module is used to extract the second attribute text information corresponding to the target attribute from the vertical field website by using the XPath final expression corresponding to the target attribute.

9. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to a vertical website information extraction method based on a large model as described in any one of claims 1-7.

10. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to a vertical website information extraction method based on a large model as described in any one of claims 1-7.

Images