Webpage data analysis method and device, computer equipment and readable storage medium

Abstract

The invention relates to a webpage data analysis method and device, computer equipment and a readable storage medium. The method comprises the steps of obtaining a to-be-processed target webpage and a first analysis rule matched with the target webpage; determining original collection data of the target webpage, and analyzing the original collection data according to a first analysis rule to obtain an analysis result; if abnormal items exist in the analysis result and the number of the abnormal items is greater than an abnormal threshold value, calling a large language model to analyze the webpage structure of the target webpage, and determining the webpage structure change of the target webpage; updating the first analysis rule according to the webpage structure change to obtain a second analysis rule; and analyzing the original collection data based on a second analysis rule to obtain target analysis data. By the adoption of the method, webpage data analysis can be achieved in a self-adaptive webpage structure change mode, and the reliability of data collection is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a web page data parsing method, apparatus, computer equipment and readable storage medium. Background Art

[0002] With the explosive growth of Internet information, more and more people are beginning to pay attention to how to find the information they need in the large amount of Internet data. Software industry professionals are also increasingly inclined to do automated processing in large amounts of data to improve information retrieval efficiency and achieve the purpose of targeted information recommendation. Web page information intelligent collection, as the main way to obtain web page information, is also known to more and more people.

[0003] However, the traditional intelligent collection of web page information often fails when the web page structure changes, and requires frequent manual intervention and adjustment. The prior art lacks a web page data analysis method that can adapt to the changes in web page structure. Summary of the invention

[0004] Based on this, it is necessary to provide a web page data parsing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can adapt to changes in web page structure in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a web page data parsing method, comprising:

[0006] Acquire a target webpage to be processed and a first parsing rule matching the target webpage;

[0007] Determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule to obtain a parsing result;

[0008] If there are abnormal items in the analysis result, and the number of the abnormal items is greater than the abnormal threshold, calling the large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage;

[0009] The first parsing rule is updated according to the webpage structure change to obtain a second parsing rule;

[0010] The original collected data is parsed based on the second parsing rule to obtain target parsed data.

[0011] In one embodiment, calling a large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage includes:

[0012] Calling a large language model to analyze the webpage structure of the target webpage to determine webpage elements in the target webpage; the webpage elements include at least one of HTML tags, CSS styles, JavaScript logic, page layout, and content organization;

[0013] If the web page elements are changed, it is determined that the web page structure of the target web page is changed, and the web page structure change of the target web page is determined according to the changed web page elements.

[0014] In one embodiment, obtaining a target webpage to be processed and a first parsing rule matching the target webpage includes:

[0015] Get the target web page to be processed;

[0016] Analyzing web page elements of the target web page based on the large language model web page analysis to obtain attribute information of the web page elements; the attribute information at least includes an identifier, an attribute value, content and layout;

[0017] A first parsing rule matching the attribute information is generated using a large language model.

[0018] In one embodiment, the updating of the first parsing rule according to the webpage structure change to obtain the second parsing rule includes:

[0019] According to the target web page element corresponding to the web page structure change, a sub-parsing rule corresponding to the target web page element is determined from the first parsing rule, and the sub-parsing rule is updated to obtain a second parsing rule.

[0020] In one embodiment, the method further comprises:

[0021] Acquire the historical original collected data, historical parsing rules and historical parsing data of the target webpage; the historical parsing data is determined by parsing the historical original collected data based on the historical parsing rules;

[0022] Analyze the historical original collected data, the historical analysis rules and the historical analysis data to determine the analysis result of the historical analysis data;

[0023] The analysis result is output to the large language model to obtain an optimization method output by the large language model, and the latest parsing rule of the target web page is optimized according to the optimization method.

[0024] In one embodiment, the original collected data is HTML data, and the parsing of the original collected data based on the second parsing rule to obtain target parsed data includes:

[0025] Based on the second parsing rule, sequence annotation or information extraction is performed on the original collected data to obtain target parsed data in the target web page.

[0026] In a second aspect, the present application also provides a web page data parsing device, comprising:

[0027] A web page information intelligent collection module is used to obtain the target web page to be processed;

[0028] An initialization module, used to determine a first parsing rule that matches the target web page;

[0029] A parsing module, used for determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule, and obtaining a parsing result;

[0030] A rule adjustment module, configured to, if there are abnormal items in the analysis result and the number of the abnormal items is greater than an abnormal threshold, call a large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage; and update the first analysis rule according to the webpage structure change to obtain a second analysis rule;

[0031] The parsing module is used to parse the original collected data based on the second parsing rule to obtain target parsed data.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Acquire a target webpage to be processed and a first parsing rule matching the target webpage;

[0034] Determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule to obtain a parsing result;

[0035] If there are abnormal items in the analysis result, and the number of the abnormal items is greater than the abnormal threshold, calling the large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage;

[0036] The first parsing rule is updated according to the webpage structure change to obtain a second parsing rule;

[0037] The original collected data is parsed based on the second parsing rule to obtain target parsed data.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0039] Acquire a target webpage to be processed and a first parsing rule matching the target webpage;

[0040] Determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule to obtain a parsing result;

[0041] If there are abnormal items in the analysis result, and the number of the abnormal items is greater than the abnormal threshold, calling the large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage;

[0042] The first parsing rule is updated according to the webpage structure change to obtain a second parsing rule;

[0043] The original collected data is parsed based on the second parsing rule to obtain target parsed data.

[0044] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0045] Acquire a target webpage to be processed and a first parsing rule matching the target webpage;

[0046] Determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule to obtain a parsing result;

[0047] If there are abnormal items in the analysis result, and the number of the abnormal items is greater than the abnormal threshold, calling the large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage;

[0048] The first parsing rule is updated according to the webpage structure change to obtain a second parsing rule;

[0049] The original collected data is parsed based on the second parsing rule to obtain target parsed data.

[0050] The web page data parsing method, apparatus, computer equipment, computer-readable storage medium and computer program product described above obtain a target web page to be processed and a first parsing rule that matches the target web page; determine the original collected data of the target web page, parse the original collected data according to the first parsing rule, and obtain a parsing result; in the case of parsing anomalies, analyze the web page structure through a large language model, generate parsing rules, and parse the original collected data, so as to detect and adapt to web page changes, realize self-adjustment and continuous optimization of intelligent collection of web page information, greatly reduce the need for manual intervention, and utilize the powerful predictive understanding ability of the large language model to reduce dependence on a large amount of pre-collected question and answer data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A diagram showing an application environment of a web page data parsing method in one embodiment;

[0053] Figure 2 A schematic diagram of a flow chart of a web page data parsing method in one embodiment;

[0054] Figure 3 A schematic diagram of a flow chart of a method for generating parsing rules in an embodiment;

[0055] Figure 4 A schematic diagram of a flow chart of an optimization method for parsing web page data in one embodiment;

[0056] Figure 5 A schematic diagram of a process of parsing web page data in one embodiment;

[0057] Figure 6 is a structural block diagram of a web page data parsing system in one embodiment;

[0058] Figure 7 is a structural block diagram of a web page data parsing device in one embodiment;

[0059] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] For different business scenarios, for example, scenarios where long-term and continuous collection of specific website content is required, such as news monitoring, product price tracking, etc.; another example is the need to collect web page content with diverse structures and frequent changes, such as social media platforms, e-commerce websites, etc.; another example is the need to collect large-scale collection of multiple websites and high manual maintenance costs.

[0062] Traditional web page information intelligent collection often fails when facing web page structure changes, requiring frequent manual intervention and adjustment. To address this problem, related technologies use robust selectors, automated testing and other methods, but this approach still cannot completely solve this problem. To address this technical problem, a method is proposed to use the capabilities of a large language model to build an intelligent parsing method that can adapt to web page structure changes, self-adjust and continuously optimize.

[0063] The web page data parsing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage device can store data that the server 104 needs to process. The data storage device can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal obtains the target web page to be processed and the first parsing rule that matches the target web page from the server; determines the original collected data of the target web page, and parses the original collected data according to the first parsing rule to obtain the parsing result; if there are abnormal items in the parsing result, and the number of abnormal items is greater than the abnormal threshold, the large language model is called to analyze the web page structure of the target web page to determine the web page structure change of the target web page; the first parsing rule is updated according to the web page structure change to obtain the second parsing rule; the original collected data is parsed based on the second parsing rule to obtain the target parsed data.

[0064] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed device consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0065] In an exemplary embodiment, Figure 2 As shown, a web page data parsing method is provided, and the method is applied to Figure 1 The terminal in is taken as an example to illustrate, including the following steps 202 to 210. Among them:

[0066] Step 202: Obtain a target webpage to be processed and a first parsing rule matching the target webpage.

[0067] The target web page to be processed may be one or more, and the number of target web pages may be determined according to actual business needs. The parsing rule may be determined by one or more of the identifier of the web page element (such as HTML tags, class names and IDs of elements), attribute values ​​of web page elements, content (such as text content and multimedia resources), layout (such as document objects, model tree structures and CSS styles) and JavaScript. The first parsing rule may include at least one of XPath, CSS selectors or regular expressions.

[0068] Exemplarily, a target web page to be processed is obtained, and the web page structure of the target web page is analyzed using a large language model (LLM), that is, an LLM web page analyzer is used to use natural language processing technology to analyze the hypertext markup language structure (HTML structure), CSS style, and JavaScript code of the target web page, and based on the analysis results, a suitable parsing rule, namely, a first parsing rule, is generated.

[0069] Step 204, determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule, and obtaining a parsing result.

[0070] The original collected data may be HTML data.

[0071] Exemplarily, data is collected on the target web page to obtain the original collected data of the target web page. The original collected data is parsed according to the first parsing rule, the HTML element containing the target data is located, and the associated data containing the target data is extracted according to the located element, and irrelevant content such as HTML tags, JavaScript codes, CSS styles, etc. in the associated data is removed, and plain text or other required formats of data are retained. It is understandable that in the case where the structure of the web page changes, when the original collected data is parsed according to the first parsing rule, there will be a situation where the element corresponding to the target data cannot be located, or the corresponding target data cannot be obtained, that is, the obtained data is a null value, in which case a parsing exception will occur. The specific parsing method can also be implemented by existing methods, which will not be elaborated here.

[0072] Step 206: If there are abnormal items in the analysis result, and the number of abnormal items is greater than the abnormal threshold, the large language model is called to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage.

[0073] The abnormal items may include but are not limited to one or more of the following: the parsing result is empty, data is missing in the parsing result, and the data format is abnormal.

[0074] The large language model is a fine-tuned model, where the fine-tuning training method includes: obtaining a large number of web page samples containing various HTML structures and elements, and annotating the key information to be extracted from the web page samples. The web page samples can be HTML pages covering different websites, different layouts, and different complexities. The HTML parsing task is clearly defined as a sequence labeling or information extraction problem. For example, the model can be allowed to recognize and mark specific HTML elements or attributes.

[0075] Use the prepared dataset (i.e. web page samples) to fine-tune the preset large language model. Fine-tuning training may include the following:

[0076] Add HTML tags and web page structure information to the input web page samples, and let the preset large language model learn the syntax of HTML and the hierarchical structure of the web page. Design specific prompts to guide the preset large language model to focus on important elements in HTML. Use multi-task learning to train the model to complete HTML parsing and related web information extraction tasks at the same time. Design appropriate evaluation indicators to measure the model's HTML parsing ability, such as accuracy, recall rate, F1 score, etc. Iterative optimization: According to the performance of the model on the test set, continuously adjust the training strategy and model parameters to improve the parsing accuracy.

[0077] Furthermore, if there is a demand for a specific type of website, the large language model can be further fine-tuned based on the web page samples of that specific type. In addition, the output of the large language model can be combined with some predefined HTML parsing rules to improve the robustness of the parsing. And / or, implement a feedback mechanism to enable the model to learn and improve from errors in the actual intelligent collection process of web page information. For the need to process multilingual web pages, multilingual HTML parsing training can be performed. In addition, with regard to the attention mechanism: strengthen the model's attention to key elements in the HTML structure, such as specific class, id or data attributes.

[0078] Exemplarily, if there are abnormal items in the parsing result, and the number of abnormal items is greater than the abnormal threshold, the large language model is called to understand the webpage structure of the target webpage, understand the structure and content of the webpage, and compare the understood structure and content with the structure and content determined in the previous analysis. If they are inconsistent, it is determined that the structure of the target webpage has changed, and the second parsing rule can be generated directly based on the understood structure and content, or the webpage structure change of the target webpage can be determined, and the second parsing rule is determined based on the webpage structure change. The webpage structure change can be, but is not limited to, at least one of a page layout change and a content organization change.

[0079] Step 208: Update the first parsing rule according to the change in the webpage structure to obtain a second parsing rule.

[0080] Exemplarily, the large language model is called to update the first parsing rule according to the webpage structure change to obtain the second parsing rule. The updating method may be to replace the abnormal sub-rule in the first parsing rule, or to directly determine the generated new parsing rule as the second parsing rule.

[0081] Step 210: parse the original collected data based on the second parsing rule to obtain target parsed data.

[0082] Exemplarily, the original collected data is parsed based on the second parsing rule. If the format of the original collected data is not the preset data format, the data format of the original collected data is converted to obtain the collected data corresponding to the preset data format. The target data is extracted from the converted collected data according to the second parsing rule to obtain the target parsed data. For example, if the format of the original collected data is JSON, a library such as Gson or FastJson can be used to parse the JSON string into a corresponding preset data structure. Further, the obtained target parsed data is displayed, and the target parsed data can be but is not limited to data such as text, pictures or links corresponding to the target object. Among them, LLM-assisted data parsing can process and structure the collected information more quickly.

[0083] The web page data parsing method obtains a target web page to be processed and a first parsing rule that matches the target web page; determines the original collected data of the target web page, parses the original collected data according to the first parsing rule, and obtains a parsing result. In the case of parsing anomalies, the web page structure is analyzed by a large language model, and parsing rules are generated to parse the original collected data, so that the web page changes can be detected and adapted to, and the rules are intelligently generated and adjusted, which reduces invalid or redundant collection operations, realizes self-adjustment and continuous optimization of intelligent collection of web page information, and greatly reduces the need for manual intervention, that is, it is easy to adapt to new website structures and content types, without a lot of reprogramming, and can be quickly deployed to new fields or application scenarios, and the powerful predictive understanding ability of the large language model is utilized to reduce dependence on a large amount of pre-collected question and answer data.

[0084] It is understandable that in actual application scenarios, the structure of a web page and the type of web page content may also be adjusted according to actual needs. In an exemplary embodiment, calling a large language model to analyze the web page structure of a target web page and determining the web page structure changes of the target web page includes the following:

[0085] Calling the large language model to analyze the webpage structure of the target webpage and determine the webpage elements in the target webpage; the webpage elements include at least one of HTML tags, CSS styles, JavaScript logic, page layout and content organization;

[0086] If the web page elements are changed, it is determined that the web page structure of the target web page is changed, and the web page structure change of the target web page is determined according to the changed web page elements.

[0087] Exemplarily, the large language model here is a trained model. Based on the large language model, the web page structure of the target web page can be analyzed and the web page content can be understood to determine the web page elements in the target web page. If the web page elements change, it is determined that the web page structure of the target web page has changed. The web page structure change of the target web page is determined based on the changed web page elements. The web page elements in the same position of the historical web page and the target web page are compared. If they are different, it is determined that the web page structure has changed, and the web page elements in the target web page are determined as the web page structure change.

[0088] In this embodiment, the large language model can be used to automatically identify the structure of a web page, determine whether the structure of the web page has changed, and then adaptively handle the changes in the structure of the web page. Compared with large-scale collection of multiple websites, changes in the structure of the web page cause the intelligent collection of web page information to fail, and frequent manual intervention is required to ensure the effectiveness of data collection and avoid waste of resources.

[0089] When collecting web page structures, if it is the first time to collect, it is necessary to analyze the web pages to generate initial parsing rules. In an exemplary embodiment, Figure 3 As shown, a method for generating parsing rules is provided, including steps 302 to 306, wherein:

[0090] Step 302: Obtain the target web page to be processed.

[0091] The number of target web pages can be one or more.

[0092] Exemplarily, in response to the data extraction instruction, the target web page to be processed corresponding to each web page identifier is obtained according to the multiple web page identifiers carried in the data extraction instruction.

[0093] Step 304: Analyze the web page elements of the target web page based on the large language model web page to obtain attribute information of the web page elements; the attribute information at least includes identifiers, attribute values, content and layout.

[0094] Among them, web page elements can be but not limited to one or more of HTML tags, CSS styles, JavaScript logic, page layout and content organization. Attribute information can be but not limited to one or more of identifiers, attribute values, content and layout. Each web page element has corresponding attribute information.

[0095] Step 306: Generate a first parsing rule matching the attribute information using the large language model.

[0096] Exemplarily, the web page elements of the target web page are analyzed based on the large language model web page to determine the attribute information corresponding to each web page element. For example, the LLM web page analyzer uses natural language processing technology to analyze the HTML structure, CSS style and JavaScript code of the web page, and the rule generator uses LLM to generate suitable parsing rules such as XPath, CSS selector or regular expression based on the analysis results.

[0097] In the above method, LLM is used in the initialization phase to analyze the target web page structure and generate initial collection rules. The semantic understanding ability of LLM helps to more accurately locate and extract the required information, ensuring the reliability of data collection.

[0098] Optionally, in an exemplary embodiment, the first parsing rule is updated according to the webpage structure change to obtain the second parsing rule, including:

[0099] According to the target web page element corresponding to the web page structure change, a sub-parsing rule corresponding to the target web page element is determined from the first parsing rule, and the sub-parsing rule is updated to obtain a second parsing rule.

[0100] Exemplarily, in the case of parsing anomaly, i.e., the web page structure changes, the large language model is directly called to locate the abnormal position of the parsing anomaly, and the corresponding target web page element is determined from the web page structure according to the abnormal position, and the attribute information of the target web page element is determined, and the sub-parsing rule corresponding to the target web page element in the first parsing rule is updated according to the attribute information to obtain the second parsing rule. Alternatively, in the case of parsing anomaly, i.e., the web page structure changes, the large language model is directly called to generate the second parsing rule corresponding to the target web page.

[0101] This approach uses a large language model to automatically analyze web page structure, generate and adjust collection rules, greatly reducing the workload of manual maintenance and updating of intelligent web information collection, as well as reducing dependence on professional web information intelligent collection engineers and reducing labor costs. In addition, LLM can intelligently analyze changes in web page structure and quickly generate new collection rules, automatically adapting to the structure and style of different websites, thereby enhancing the versatility and robustness of intelligent web information collection.

[0102] Furthermore, the original collected data is HTML data, and the original collected data is parsed based on the second parsing rule to obtain target parsed data, including: based on the second parsing rule, sequence tagging or information extraction of the original collected data to obtain target parsed data in the target web page.

[0103] Among them, the parsing rules clearly state which specific information needs to be extracted from the web page.

[0104] Exemplarily, the original collected HTML data is preprocessed, for example, unnecessary tags, scripts, styles, etc. are removed, and only the parts related to the target information are retained to obtain the preprocessed data, and the preprocessed data is sequence-annotated and then information is extracted to obtain the target parsed data in the target web page. Sequence annotation and information extraction can be implemented by existing methods, which will not be described in detail here. This method parses the original collected data based on the updated parsing rules to enhance the accuracy and reliability of the collected data.

[0105] It is understandable that the existing web page information intelligent collection device lacks the ability of self-adjustment and continuous optimization. In order to achieve self-adjustment and continuous optimization, in an exemplary embodiment, Figure 4 As shown, a method for optimizing web page data parsing is provided, including steps 402 to 406, wherein:

[0106] Step 402, obtaining the historical original collected data, historical parsing rules and historical parsing data of the target webpage; the historical parsing data is determined by parsing the historical original collected data based on the historical parsing rules.

[0107] Exemplarily, historical original collected data, historical parsing rules and historical parsing data of the target web page are periodically acquired.

[0108] Step 404: Analyze the historical original collected data, the historical analysis rules and the historical analysis data to determine the analysis results of the historical analysis data.

[0109] Step 406 outputs the analysis result to the large language model, obtains an optimization method output by the large language model, and optimizes the latest parsing rules of the target web page according to the optimization method.

[0110] The analysis result may be, but is not limited to, the completeness of the historically collected data, and the data analysis may be achieved through an analyzer that integrates data analysis, which is not limited here. The latest parsing rule may be based on the second parsing rule above, after the original collected data is parsed, if the target web page does not change the result, then the second parsing rule may be confirmed as the latest parsing rule.

[0111] For example, by analyzing the historical original collected data, historical parsing rules and historical collected data, the historical collected data of the target web page A in the past week is obtained, and 5% of the field content extraction is incomplete, that is, the completeness of the historical collected data is 95%, which is not in the preset completeness range of 98% to 100%. 5% of the field content extraction is incomplete and output to the large language model, and an optimized method of the large language model output is obtained, that is, adding an additional backup selector for content extraction.

[0112] In the above embodiment, historical collection data is analyzed regularly so that LLM can generate an optimized method for parsing rules according to the analysis results, continuously optimize the parsing rules, and continuously improve its collection strategy and efficiency over time, thereby realizing continuous self-optimization of the intelligent web information collection device.

[0113] In an exemplary embodiment, a web page data parsing method is provided, wherein the method is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0114] Obtain a target web page to be processed and a first parsing rule matching the target web page; determine the original collected data of the target web page, parse the original collected data according to the first parsing rule, and obtain a parsing result; if there are abnormal items in the parsing result, and the number of abnormal items is greater than the abnormal threshold, call a large language model to analyze the web page structure of the target web page, and determine the web page structure change of the target web page; update the first parsing rule according to the web page structure change to obtain a second parsing rule; parse the original collected data based on the second parsing rule to obtain target parsed data.

[0115] Periodically obtain historical collection data and historical parsing rules corresponding to the historical collection data; analyze the historical collection data and historical parsing rules through a large language model to determine the target factors that affect the collection performance; determine the corresponding collection optimization method based on the target factors, and adjust the corresponding network environment based on the collection optimization method.

[0116] It should be noted that the specific implementation of this embodiment can be achieved through the above-mentioned limitation method, which will not be elaborated here.

[0117] In an exemplary embodiment, Figure 5 As shown, a flow chart of web page data parsing is provided, including: obtaining a target web page to be processed, if the target web page is collected for the first time, then collecting for the first time, analyzing the web page through a large language model and generating an initial parsing rule, i.e., a first parsing rule, and then performing a collection operation; if it is not the first collection operation, directly obtaining the first parsing rule corresponding to the target web page, and then performing a collection operation, parsing the original collected data obtained by the collection using the determined parsing rule, if the parsing is normal, obtaining target parsed data; if the parsing is abnormal and the number of abnormalities reaches a threshold, calling the large language model to analyze the structure of the target web page and generating a new parsing rule, i.e., a second parsing rule, parsing the original collected data according to the second parsing rule, obtaining target parsed data, and storing the target parsed data.

[0118] Furthermore, historical collection data and historical parsing rules corresponding to the historical collection data are obtained; the historical collection data and historical parsing rules are analyzed through a large language model to determine the target factors that affect the collection performance; the corresponding collection optimization method is determined according to the target factors, and the corresponding network environment is adjusted based on the collection optimization method.

[0119] This embodiment uses LLM to analyze the target web page structure and generate initial parsing rules in the initialization stage; performs collection and uses LLM to interpret the extracted data in the data collection stage; uses LLM to analyze the problem and generate new collection rules when the collection fails; and regularly uses LLM to analyze the collection history and make optimization suggestions in the continuous optimization stage. This method solves the collection failure problem caused by changes in web page structure, significantly improves the adaptability of web page information intelligent collection to web page changes, greatly reduces the need for manual intervention, ensures the continuous availability of data, can be applied in application scenarios that require real-time or frequent collection, and at the same time enhances the accuracy and reliability of the collected data, realizes the continuous self-optimization of the web page information intelligent collection device, and provides an intelligent and adaptive solution for the field of network data collection.

[0120] In addition, it has improved the intelligence level of web page collection, brought significant technological progress to the field of data collection and analysis, and is conducive to improving the data-driven decision-making capabilities of various industries.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a web page data parsing system for implementing the web page data parsing method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more web page data parsing system embodiments provided below can refer to the limitations of the web page data parsing method above, and will not be repeated here.

[0123] In an exemplary embodiment, Figure 6 As shown, a web page data parsing system is provided, including: a central processing unit, an initialization module, a parsing module, a rule adjustment module, an optimization module, an external interface and a large language model application programming interface (Large Language Model Application Programming Interface, LLMAPI), wherein the central processing unit can also be called an LLM integrated web page information intelligent acquisition controller, the initialization module includes an LLM web page analyzer and a rule generator, the LLM web page analyzer is used to analyze the target web page, and the rule generator is used to generate corresponding parsing rules based on the analysis results of the web page.

[0124] The parsing module includes a data extractor for parsing the collected raw data. The rule adjustment module includes an error detector and an LLM rule adjuster. The error detector is used to detect whether there are any abnormalities in the parsing. When there are abnormalities and the number of abnormalities is greater than the abnormality threshold, the old parsing rules are adjusted by the LLM rule adjuster to generate new parsing rules to complete the parsing of the original collected data.

[0125] The optimization module includes a historical analyzer and an LLM rule optimizer. The historical analyzer collects and organizes past acquisition data and performance indicators, and the LLM rule optimizer analyzes historical data and proposes suggestions for improving acquisition strategies. External interfaces include a target web page input interface, an acquisition data storage interface, and a manual supervision interface. The LLM API is used to connect to all components that use LLM.

[0126] The web page data analysis system mentioned above uses a large language model (LLM) to analyze the structure of web pages and generate collection rules, so that the system can automatically adapt to changes in web pages. Through the powerful language understanding ability of LLM, it reduces the dependence on a large amount of pre-collected question and answer data, realizes self-adjustment and continuous optimization of intelligent web page information collection, greatly reduces the need for manual intervention, and uses the reasoning and analysis capabilities of LLM to improve the system's ability to handle complex and real-time problems, as well as improve the accuracy of data extraction and analysis. In addition, by regularly using LLM to analyze the collection history and make optimization suggestions, the system can achieve self-improvement, avoiding the problem that the system performance may decline over time and cannot adapt to the dynamically changing network environment.

[0127] Based on the same inventive concept, the embodiment of the present application also provides a web page data parsing device for implementing the web page data parsing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more web page data parsing device embodiments provided below can refer to the limitations of the web page data parsing method above, and will not be repeated here.

[0128] In an exemplary embodiment, Figure 7 As shown, a web page data parsing device is provided, including: an acquisition module 702, an initialization module 704, a parsing module 706, and a rule adjustment module 708, wherein:

[0129] The acquisition module 702 is used to acquire the target web page to be processed; the initialization module 704 is used to determine the first parsing rule that matches the target web page.

[0130] The parsing module 706 is used to determine the original collected data of the target webpage, and parse the original collected data according to the first parsing rule to obtain a parsing result.

[0131] The rule adjustment module 708 is used to call the large language model to analyze the web page structure of the target web page if there are abnormal items in the analysis results and the number of abnormal items is greater than the abnormal threshold, and determine the web page structure changes of the target web page; update the first analysis rule according to the web page structure changes to obtain the second analysis rule.

[0132] The parsing module 706 is used to parse the original collected data based on the second parsing rule to obtain target parsed data.

[0133] The web page data parsing device obtains a target web page to be processed and a first parsing rule that matches the target web page; determines the original collected data of the target web page, parses the original collected data according to the first parsing rule, and obtains a parsing result. In the case of parsing anomalies, the web page structure is analyzed by a large language model, and parsing rules are generated to parse the original collected data, so that the web page changes can be detected and adapted to, thereby realizing self-adjustment and continuous optimization of intelligent collection of web page information, greatly reducing the need for manual intervention, and utilizing the powerful predictive understanding ability of the large language model to reduce dependence on a large amount of pre-collected question and answer data.

[0134] In an exemplary embodiment, the detection module 708 is further used to call the large language model to analyze the webpage structure of the target webpage to determine the webpage elements in the target webpage; the webpage elements include at least one of HTML tags, CSS styles, JavaScript logic, page layout and content organization;

[0135] If the web page elements are changed, it is determined that the web page structure of the target web page is changed, and the web page structure change of the target web page is determined according to the changed web page elements.

[0136] In an exemplary embodiment, the initialization module 704 is further used to obtain a target web page to be processed;

[0137] Analyzing web page elements of a target web page based on a large language model web page to obtain attribute information of the web page elements; the attribute information at least includes an identifier, an attribute value, content, and layout;

[0138] A first parsing rule matching the attribute information is generated using a large language model.

[0139] In an exemplary embodiment, the rule adjustment module 708 is further used to determine a sub-parsing rule corresponding to the target web page element corresponding to the web page structure change from the first parsing rule, update the sub-parsing rule, and obtain a second parsing rule.

[0140] In an exemplary embodiment, the web page data parsing device further includes an optimization module, the optimization module is used to obtain the historical original collected data, the historical parsing rules and the historical parsing data of the target web page; the historical parsing data is determined by parsing the historical original collected data based on the historical parsing rules;

[0141] Analyze the historical original collected data, historical analysis rules and historical analysis data to determine the analysis results of the historical analysis data;

[0142] The analysis results are output to the large language model to obtain an optimization method for the large language model output, and the latest parsing rules of the target web page are optimized according to the optimization method.

[0143] In an exemplary embodiment, the parsing module 706 is further configured to perform sequence annotation or information extraction on the original collected data based on the second parsing rule to obtain target parsed data in the target web page.

[0144] Each module in the web page data parsing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0145] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a device bus, and the communication interface, the display unit and the input device are connected to the device bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device and a computer program. The internal memory provides an environment for the operation of the operating device and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a web page data parsing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0146] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0148] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0151] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A web page data parsing method, characterized in that: The method comprises: Acquire a target webpage to be processed and a first parsing rule matching the target webpage; Determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule to obtain a parsing result; If there are abnormal items in the analysis result, and the number of the abnormal items is greater than the abnormal threshold, calling the large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage; The first parsing rule is updated according to the webpage structure change to obtain a second parsing rule; The original collected data is parsed based on the second parsing rule to obtain target parsed data.

2. The method according to claim 1, characterized in that The calling of the large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage includes: Calling a large language model to analyze the webpage structure of the target webpage to determine webpage elements in the target webpage; the webpage elements include at least one of HTML tags, CSS styles, JavaScript logic, page layout, and content organization; If the web page elements are changed, it is determined that the web page structure of the target web page is changed, and the web page structure change of the target web page is determined according to the changed web page elements.

3. The method according to claim 1, characterized in that The step of obtaining a target webpage to be processed and a first parsing rule matching the target webpage includes: Get the target web page to be processed; Analyzing web page elements of the target web page based on the large language model web page to obtain attribute information of the web page elements; the attribute information at least includes an identifier, an attribute value and content; A first parsing rule matching the attribute information is generated using a large language model.

4. The method according to claim 2, characterized in that: The updating of the first parsing rule according to the webpage structure change to obtain the second parsing rule includes: According to the target web page element corresponding to the web page structure change, a sub-parsing rule corresponding to the target web page element is determined from the first parsing rule, and the sub-parsing rule is updated to obtain a second parsing rule.

5. The method according to claim 1, characterized in that The method further comprises: Acquire the historical original collected data, historical parsing rules and historical parsing data of the target webpage; the historical parsing data is determined by parsing the historical original collected data based on the historical parsing rules; Analyze the historical original collected data, the historical analysis rules and the historical analysis data to determine the analysis result of the historical analysis data; The analysis result is output to the large language model to obtain an optimization method output by the large language model, and the latest parsing rule of the target web page is optimized according to the optimization method.

6. The method according to any one of claims 1 to 5, characterized in that: The original collected data is HTML data, and the parsing of the original collected data based on the second parsing rule to obtain target parsed data includes: Based on the second parsing rule, sequence annotation or information extraction is performed on the original collected data to obtain target parsed data in the target web page.

7. A web page data parsing device, characterized in that: The device comprises: A data acquisition module, used to acquire a target web page to be processed; An initialization module, used to determine a first parsing rule that matches the target web page; A parsing module, used for determining the original collected data of the target webpage, parsing the original collected data according to the first parsing rule, and obtaining a parsing result; A rule adjustment module, configured to, if there are abnormal items in the analysis result and the number of the abnormal items is greater than an abnormal threshold, call a large language model to analyze the webpage structure of the target webpage to determine the webpage structure change of the target webpage; and update the first analysis rule according to the webpage structure change to obtain a second analysis rule; The parsing module is used to parse the original collected data based on the second parsing rule to obtain target parsed data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images