Effectiveness detection method and device for extraction rule, equipment and storage medium

Abstract

The embodiment of the invention provides an extraction rule validity detection method and device, equipment and a storage medium, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and auxiliary driving, and the method comprises the following steps: in the process that terminal equipment requests to access a target website, the terminal equipment uploads rendered webpage data to a server, and sends the rendered webpage data to the server; the problem that the website server carries out IP banning on the server is avoided, and the correctness of the obtained webpage data is ensured. According to the method, the server obtains the matched extraction rule based on the website link contained in the webpage data, then extracts the key information from the webpage data by adopting the extraction rule, and verifies the correctness of the key information to obtain the validity detection result of the extraction rule, so that the invalid extraction rule in the preset rule base is rapidly detected in real time; therefore, the invalid extraction rule can be maintained in time subsequently, the effect of filtering interference factors in the webpage data is ensured, and the use experience of a website is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for detecting the validity of an extraction rule. Background Art

[0002] With the rapid development of Internet technology, various websites have emerged one after another. Users can search for relevant websites in various websites to meet their own needs. Among them, various websites usually include: novel websites, video websites, image websites, etc.

[0003] When users search for websites through their browsers and read relevant content, the websites display not only key information but also a large number of distractions, significantly impacting the user experience. For example, when users search for novels through their browsers and read online novels, the websites display not only the novel content but also numerous advertisements, some of which even obscure the content, impacting the reading experience.

[0004] In related technologies, search servers maintain a rule library for a vast number of websites. This rule library contains numerous extraction rules for extracting key website information and filtering out interference factors. However, in practice, websites require constant updating and maintenance, and each update and maintenance effort can cause changes to the website's structure. In such cases, the corresponding extraction rules in the rule library can become invalid, impacting the effectiveness of filtering out interference factors on the website. Therefore, it is necessary to promptly detect invalid extraction rules in the rule library so that relevant personnel can maintain these invalid rules.

[0005] Under the relevant technology, the search server periodically polls to obtain web page source code from each website server and parses the web page source code into web page data. It then uses the matching extraction rules in the rule library to extract key information from the web page data, and compares the key information for validity to determine whether the obtained extraction rules are valid.

[0006] However, when the search server polls for a long time to obtain web page source code from the website server, the website server may block the search server's IP address, which will make the web page source code obtained by the search server blank, resulting in low accuracy of validity detection.

[0007] In addition, in the client rendering mode, the search server parses the web page source code to obtain the original page data (i.e., HTML structure) that has not been rendered. At this time, it is necessary to run an additional headless browser in the search server to render the original page data obtained by parsing, obtain the rendered page data (HTML page containing data and structure), and then perform validity detection of the extraction rules. This will cause excessive memory consumption of the server and low efficiency of validity detection of the extraction rules. Summary of the Invention

[0008] The embodiments of the present application provide a method, apparatus, device, and storage medium for detecting the validity of extraction rules, which are used to improve the efficiency and accuracy of detecting the validity of extraction rules and reduce the memory information of the server.

[0009] In one aspect, an embodiment of the present application provides a method for detecting the validity of an extraction rule, which is applied to a server and includes:

[0010] Receiving rendered web page data sent by a terminal device, wherein the web page data is obtained after the terminal device requests to access a target website, and the web page data at least includes a website link of the target website;

[0011] When the preset rule library contains a first extraction rule that matches the website link, extract key information from the web page data using the first extraction rule to obtain target key information;

[0012] A format check and a content check are performed on the target key information, and when at least one of the format check and the content check fails, the first extraction rule is marked as invalid.

[0013] In one aspect, an embodiment of the present application provides a device for detecting the validity of an extraction rule, which is applied to a server and includes:

[0014] a receiving module, configured to receive rendered web page data sent by a terminal device, wherein the web page data is obtained after the terminal device requests to access a target website, and the web page data at least includes a website link of the target website;

[0015] an extraction module configured to extract key information from the web page data using a first extraction rule matching the website link when a preset rule library contains the first extraction rule to obtain target key information;

[0016] The verification module is configured to perform format verification and content verification on the target key information, and mark the first extraction rule as invalid when at least one of the format verification and the content verification fails.

[0017] Optionally, an update module is also included;

[0018] The update module is specifically used for:

[0019] After marking the first extraction rule as invalid, predicting a second extraction rule corresponding to the target website based on features of key information contained in the webpage data;

[0020] Add the second extraction rule to the preset rule library.

[0021] Optionally, the extraction module is further configured to:

[0022] If the preset rule library does not include the first extraction rule that matches the website link, querying the cache area using the website link as a keyword to obtain the number of failed matches for the website link;

[0023] When the number of failed matches of the website link is greater than a preset threshold, a second extraction rule corresponding to the target website is predicted based on features of key information contained in the web page data, and the second extraction rule is added to the preset rule library.

[0024] Optionally, the extraction module is further configured to:

[0025] If the number of failed matches of the website link is not greater than the preset threshold, the number of failed matches of the website link is increased by one to obtain an updated number of failed matches of the website link.

[0026] Optionally, the verification module is specifically configured to:

[0027] Performing format verification on the target key information based on preset text format rules;

[0028] When the format check passes, the text type corresponding to the target key information is predicted based on the text semantic features of the target key information, and the content check of the target key information is performed based on the text type.

[0029] On the one hand, an embodiment of the present application provides a device for detecting the validity of an extraction rule, which is applied to a terminal device and includes:

[0030] An acquisition module, configured to request access to a target website and obtain rendered web page data corresponding to the target website, wherein the web page data at least includes a website link of the target website;

[0031] A sending module is used to send the web page data to a server so that when the server determines that a preset rule library contains a first extraction rule that matches the website link, it uses the first extraction rule to extract key information from the web page data to obtain target key information; and performs format verification and content verification on the target key information, and when at least one of the format verification and the content verification fails, marks the first extraction rule as invalid.

[0032] Optionally, it further includes an identification module;

[0033] The identification module is specifically used for:

[0034] Before sending the web page data to the server, a site identification script is used to identify the website type of the target website based on the web page data, and the website type is confirmed to be a designated website type.

[0035] Optionally, the identification module is specifically configured to:

[0036] extracting HTML tags associated with the specified website type from the web page data using the site identification script;

[0037] Based on the text information extracted from the HTML tags, the website type of the target website is identified.

[0038] Optionally, the web page data is an HTML page rendered in a client-side rendering mode or a server-side rendering mode.

[0039] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for detecting the validity of extraction rules when executing the program.

[0040] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned extraction rule validity detection method.

[0041] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned extraction rule validity detection method.

[0042] In an embodiment of the present application, after the terminal device requests access to the target website, it obtains the rendered web page data and uploads the web page data to the server. Compared with the technical solution in which the server periodically polls to obtain the source code of the website from the website server and parses the source code into web page data, the present application avoids the problem of the website server banning the IP address of the server and ensures the correctness of the obtained web page data.

[0043] Secondly, in the client-side rendering mode, the rendered web page data uploaded by the terminal device is complete web page data. Therefore, there is no need to run an additional headless browser on the server to render the original page data to obtain complete web page data, and then perform validity detection of the extraction rules. This not only reduces the memory consumption of the server, but also improves the efficiency of the validity detection of the extraction rules.

[0044] In addition, the server obtains matching extraction rules based on the website links contained in the web page data, and then uses the extraction rules to extract key information from the web page data, and verifies the correctness of the key information to obtain the validity test results of the extraction rules, thereby realizing real-time and rapid detection of invalid extraction rules in the preset rule library, so that the invalid extraction rules can be maintained in a timely manner in the future, ensuring the effect of filtering interference factors in the web page data and improving the user experience of the website. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of the structure of a system architecture provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0049] Figure 4 A flowchart of a method for detecting the validity of an extraction rule provided in an embodiment of the present application;

[0050] Figure 5 A flowchart of a web page data filtering method provided in an embodiment of the present application;

[0051] Figure 6A flowchart of a model training and prediction method provided in an embodiment of the present application;

[0052] Figure 7 A flowchart of a format verification and content verification method provided in an embodiment of the present application;

[0053] Figure 8 A flowchart of a method for updating book source rules provided in an embodiment of the present application;

[0054] Figure 9 A flowchart of a method for updating book source rules provided in an embodiment of the present application;

[0055] Figure 10 A flowchart of a method for detecting the validity of an extraction rule provided in an embodiment of the present application;

[0056] Figure 11 A schematic diagram of the structure of a device for detecting the validity of extraction rules provided in an embodiment of the present application;

[0057] Figure 12 A schematic diagram of the structure of a device for detecting the validity of extraction rules provided in an embodiment of the present application;

[0058] Figure 13 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0061] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0062] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] The Large Language Model (LLM) is a deep learning model trained on massive amounts of text data. It can not only generate natural language text, but also deeply understand the text's meaning and handle various natural language tasks such as text summarization, question answering, and translation.

[0064] Book source rules: refers to the extraction rules for extracting key information from novel websites, usually composed of regular expressions or XPath rules. The key information in novel websites includes: text, directory links, author name, book title, etc.

[0065] XPath rules: A language for finding information in XML and HTML documents, similar to regular expressions.

[0066] A web page is a document written in languages ​​like HTML, CSS, and JavaScript. It is the basic unit of a website. Each web page has a unique URL and can be accessed through the internet.

[0067] A website is a collection of multiple web pages, typically including a homepage, navigation pages, content pages, and contact pages, all interconnected by hyperlinks. A website typically has a separate domain name and hosting space, and provides specific services and functionality. Simply put, a webpage is a single page within a website, while a website is a collection of multiple web pages.

[0068] Client-side rendering: A rendering method in which when a client requests a web page, the server receives empty HTML data. The server then returns an initial response containing the HTML structure and JavaScript code. The client then executes the JavaScript code based on the HTML structure. During execution, the client interacts with the server via AJAX or RESTful APIs, retrieves the data, and then dynamically renders it on the client, resulting in a complete HTML page.

[0069] Server-side rendering: A rendering method in which, when a client requests a web page, the server executes JavaScript code based on the HTML structure to generate a complete HTML page, which contains both data and structure. The server then sends this HTML page directly to the client, which parses the HTML page to display the page content.

[0070] Redis (Remote Dictionary Server), also known as remote dictionary service, is an open source log-type key-value database written in ANSI C language, supports networking, and can be memory-based or persistent.

[0071] Memcached: A memory caching system that focuses on key-value pair storage. It has a simple design and is primarily used to cache frequently used data, making it suitable for distributed environments.

[0072] Ehcache database: An open source cache library widely used in Java applications. It can be embedded in applications as a local cache or integrated into a distributed cache system.

[0073] ChatGLM-6B is an open-source conversational language model that supports bilingual question-answering in Chinese and English and is optimized for Chinese. Based on the General Language Model (GLM) architecture, this model has 6.2 billion parameters. Combined with model quantization technology, it can be deployed locally on consumer-grade graphics cards.

[0074] The following is an introduction to the design concept of the embodiments of the present application.

[0075] When users search for websites through their browsers and read relevant content, the websites display not only key information but also a large number of distractions, significantly impacting the user experience. For example, when users search for novels through their browsers and read online novels, the websites display not only the novel content but also numerous advertisements, some of which even obscure the content, impacting the reading experience.

[0076] In related technologies, a search server corresponding to a browser maintains a rule library for a vast number of websites. This rule library contains a large number of extraction rules for extracting key website information. When a user searches for a website through a browser, the search server retrieves the website's webpage data and then extracts the key information from the webpage data using the matching extraction rules in the rule library. At the same time, it filters out interference factors other than the key information from the webpage data and then returns the key information to the browser for display, thereby improving the website user experience.

[0077] However, in practice, websites require constant updating and maintenance. Each time a website is updated and maintained, the structure of the web pages within it changes accordingly. In this case, the corresponding extraction rules in the rule base become invalid, affecting the effectiveness of filtering out interference factors in web data. Therefore, it is necessary to promptly detect invalid extraction rules in the rule base so that relevant personnel can maintain them.

[0078] Under the relevant technology, the search server periodically polls to obtain web page source code from each website server and parses the web page source code into web page data. It then uses the matching extraction rules in the rule library to extract key information from the web page data, and compares the key information for validity to determine whether the obtained extraction rules are valid.

[0079] However, when the search server polls for a long time to obtain web page source code from the website server, the website server may block the search server's IP address, which will make the web page source code obtained by the search server blank, resulting in low accuracy of validity detection.

[0080] In addition, in the client rendering mode, the search server parses the web page source code to obtain the original page data (i.e., HTML structure) that has not been rendered. At this time, it is necessary to run an additional headless browser in the search server to render the original page data obtained by parsing, obtain the rendered page data (HTML page containing data and structure), and then perform validity detection of the extraction rules. This will cause excessive memory consumption of the server and low efficiency of validity detection of the extraction rules.

[0081] In view of this, an embodiment of the present application provides a method for detecting the validity of an extraction rule, the method comprising the following steps:

[0082] After requesting access to a target website, the terminal device obtains rendered web page data and then uploads the web page data to a server. The web page data includes at least a website link for the target website. The server queries a rule library based on the website link for the target website. When a preset rule library contains a first extraction rule that matches the website link, the server uses the first extraction rule to extract key information from the web page data to obtain the target key information. The target key information is then subjected to format and content verification. If at least one of the format and content verifications fails, the first extraction rule is marked as invalid.

[0083] In an embodiment of the present application, after the terminal device requests to access the target website, it obtains the rendered web page data and uploads the web page data to the server. That is, multiple terminal devices access the target website in a distributed manner and parse to obtain the web page data. Compared with the technical solution in which a single-point server periodically polls to obtain the source code of the website from the website server and parses the source code into web page data, the present application avoids the problem of the website server banning the IP address of the single-point server, and ensures the correctness of the obtained web page data.

[0084] Secondly, in the client-side rendering mode, the rendered web page data uploaded by the terminal device is complete web page data. Therefore, there is no need to run an additional headless browser on the server to render the original page data to obtain complete web page data, and then perform validity detection of the extraction rules. This not only reduces the memory consumption of the server, but also improves the efficiency of the validity detection of the extraction rules.

[0085] In addition, the server obtains matching extraction rules based on the website links contained in the web page data, and then uses the extraction rules to extract key information from the web page data, and verifies the correctness of the key information to obtain the validity test results of the extraction rules, thereby realizing real-time and rapid detection of invalid extraction rules in the preset rule library, so that the invalid extraction rules can be maintained in a timely manner in the future, ensuring the effect of filtering interference factors in the web page data and improving the user experience of the website.

[0086] The following is a brief introduction to the system architecture diagram to which the technical solution of the embodiment of the present application can be applied. It should be noted that the system architecture diagram introduced below is only used to illustrate the embodiment of the present application and is not limiting.

[0087] refer to Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application. The system architecture includes at least a terminal device 101, a server 102 and a website service end 103. The number of terminal devices 101 can be one or more, the number of servers 102 can be one or more, and the number of website service ends 103 can be one or more. The present application does not specifically limit the number of terminal devices 101, servers 102 and website service ends 103.

[0088] The terminal device 101 is pre-installed with a browser application, where the browser application is a client application, a web application, a mini-program application, etc. The terminal device 101 can be a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, an intelligent voice interaction device, an intelligent vehicle-mounted device, etc., but is not limited thereto.

[0089] Server 102 is the backend server of the browser application. Server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 101 and server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0090] The website server 103 is a server for providing website source code. The website server 103 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server 102 and the website server 103 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0091] In actual applications, a preset rule library is maintained in server 102, which includes a large number of extraction rules for extracting key information from web pages. When the browser activates the interference factor filtering function and the user searches for a website through the browser, terminal device 101 sends a search request to server 102. Based on the search request, server 102 obtains the corresponding website source code from website server 103 and then parses the website source code into web page data. The key information in the web page data is extracted using the matching extraction rules in the preset rule library. At the same time, interference factors other than the key information in the web page data are filtered out, and the key information is then returned to the browser for display.

[0092] In addition, when the user searches for a website and browses the website content through a browser, regardless of whether the browser has enabled the interference factor filtering function, the terminal device will upload the rendered web page data to the server 102. Based on the received web page data, the server 102 determines whether the matching extraction rules in the preset rule library are valid, so as to promptly maintain the invalid extraction rules.

[0093] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0094] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limiting.

[0095] Application scenario 1: Novel website scenario.

[0096] See also Figure 2 , a book source rule library is maintained in the server, and the book source rule library includes a large number of book source rules for extracting key information of web pages in the novel website.

[0097] When the user searches for novel website A through a browser application and browses novel web page B on novel website A, the terminal device uploads the novel web page data of novel web page B to the server. The novel web page data includes: book name "XXX", author name "abc", directory list "Chapter 1, Chapter 2,..., Chapter 6", completion status "Completed", book type "Martial Arts Novels" and advertisements.

[0098] The server obtains a book source rule C that matches the website link of the novel website A from the book source rule library, and then extracts key information from the novel web page data using the book source rule C.

[0099] Assuming that the extracted key information is empty, it means that the extracted key information is wrong, that is, the format check and content check of the key information both fail, and the book source rule C is marked as invalid.

[0100] Assuming that the extracted key information includes: book name "XXX", author name "abc", table of contents list "Chapter 1, Chapter 2, ..., Chapter 6", completion status "Completed", and book type "Martial Arts Novels", it means that the extracted key information is correct, and the book source rule C is marked as valid.

[0101] Application scenario 2: Video website scenario.

[0102] See also Figure 3 , a video rule library is maintained in the server, and the video rule library includes a large number of video rules for extracting key information in the video website.

[0103] When the user searches for video website X through a browser application and watches movie Y on video website X, the terminal device uploads the video web page data of movie Y to the server. The video web page data includes: video name "Movie Y", video type "Science Fiction", video cover image "Movie Poster", the name of the starring actor "def" and advertisements.

[0104] The server obtains a video rule F that matches the website link of the video website X from the video rule library, and then extracts key information from the video web page data using the video rule F.

[0105] Suppose the extracted key information includes a video cover image of "Sci-Fi," a video genre of "Movie Poster," a video title of "Movie Y," and a starring actor of "def." Verifying the key information reveals that the extracted video cover image and genre are incorrect. Therefore, video rule F is marked invalid.

[0106] It should be noted that the application scenarios applicable to the embodiments of the present application are not limited to the above-mentioned ones. During the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0107] Below, in combination with the system architecture and application scenarios described above, the method provided by the exemplary embodiment of the present application is described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0108] See also Figure 4 , provides a process for detecting the validity of an extraction rule in an embodiment of the present application, the process of the method is executed by a computer device, which may be Figure 1 The terminal device and the server interact with each other to execute the method. The specific implementation process of the method is as follows:

[0109] In step 401 , the terminal device requests to access a target website and obtains rendered web page data corresponding to the target website.

[0110] Specifically, a website includes multiple web pages, and the web page structures of multiple web pages within the same website are similar. In response to an access operation triggered in a browser application for a specific web page on the target website, the terminal device sends an access request to the server. The server obtains the page source code of the web page from the website server. Based on the page source code, the rendered web page data is obtained, which is a website snapshot of the target website.

[0111] In some embodiments, the web page data is rendered into an HTML page using a client-side rendering mode or a server-side rendering mode.

[0112] When using server-side rendering, the server executes JavaScript code based on the HTML structure to generate a complete HTML page, which contains data and structure. The server then sends the HTML page to the terminal device.

[0113] When using client-side rendering, the server returns an initial response containing HTML and JavaScript code to the terminal device. The browser application in the terminal device executes the JavaScript code based on the HTML structure. During execution, it interacts with the server through AJAX or RESTful APIs, retrieves the data, and dynamically renders it to produce a complete HTML page.

[0114] In an embodiment of the present application, since the terminal device obtains rendered web page data after requesting access to the target website and uploads the web page data to the server, regardless of whether the web page data is obtained by rendering in client rendering mode or server rendering mode, the server does not need to run an additional headless browser to render the original page data to obtain complete web page data. This not only reduces the memory consumption of the server, but also improves the efficiency of detecting the validity of the extraction rules.

[0115] In some embodiments, the webpage data is obtained in a scenario where a browser has enabled a noise factor filtering function, or in a scenario where the browser has not enabled the noise factor filtering function. Specifically, enabling the noise factor filtering function means extracting key information from the webpage data using matching extraction rules in a preset rule library, and filtering out noise factors other than the key information from the webpage data.

[0116] When the interference factor filtering function is activated, the following situations exist:

[0117] In the first scenario, a pre-set rule library contains valid extraction rules that match the website link. Key information is extracted from the webpage data using these extraction rules, and interference factors other than the key information are filtered out. The filtered webpage data is then returned to the terminal device. The terminal device then displays the filtered webpage data in the browser application and uploads it to the server for rule validity testing.

[0118] In case 2, if the preset rule library does not contain an extraction rule that matches the website link, or contains an extraction rule that matches the website link but is invalid, the unfiltered web page data is returned to the terminal device. The terminal device displays the unfiltered web page data in the browser application and uploads the unfiltered web page data to the server for rule validity verification.

[0119] If the interference factor filtering function is not enabled, the unfiltered web page data is returned to the terminal device. The terminal device displays the unfiltered web page data in the browser application and uploads the unfiltered web page data to the server for rule validity testing.

[0120] For example, see Figure 5 When the user searches for novel website A through a browser application and browses novel web page B in novel website A, the unfiltered novel web page data 501 includes: book name "XXX", author name "abc", directory list "Chapter 1, Chapter 2, ..., Chapter 6", completion status "Completed", book type "Martial Arts Novels" and advertisements.

[0121] The novel web page data 502 after filtering out advertisements includes: book name "XXX", author name "abc", directory list "Chapter 1, Chapter 2, ..., Chapter 6", completion status "Completed", and book type "Martial Arts Novels".

[0122] In an embodiment of the present application, the server supports the use of unfiltered or filtered web page data to perform rule validity detection, thereby realizing real-time and rapid validity detection of the extraction rules in the rule library without affecting the use of the interference factor filtering function, thereby improving the user experience of the website.

[0123] Step 402: The terminal device sends the web page data to the server.

[0124] Specifically, a site identification script is embedded in a browser application pre-installed on a terminal device. When a user accesses a target website through the browser engine and reads relevant content on a web page within the target website, the site identification script identifies the type of the target website based on the web page data. If the website type is confirmed to be a specified type, the web page data is sent to the server.

[0125] In actual applications, the designated website type can be set according to actual needs. For example, the designated website type can be a novel website, a video website, a picture website, etc.

[0126] In an embodiment of the present application, a site identification script is embedded in the browser application to identify the website type of the target website accessed by the terminal device in real time, and when the website type is identified as a specified website type, the web page data is uploaded to the server, avoiding uploading the web page data of all websites to the server, saving communication resources and server storage space, and at the same time realizing the validity detection of the extraction rules corresponding to the specified type of website, which is more targeted.

[0127] In some embodiments, a site identification script is used to extract HTML tags associated with a specified website type from web page data, and the website type of the target website is identified based on text information within the extracted HTML tags.

[0128] Specifically, the HTML tags associated with the specified website type are selected from a variety of tags in the HTML page based on characteristics of the specified website type. The HTML tags associated with the specified website type include one or more categories of HTML tags. After extracting the HTML tags associated with the specified website type from the webpage data, text information is extracted from each HTML tag, and then the website type of the target website is determined based on the textual semantic features of the text information.

[0129] For example, a site identification script embedded in a browser application extracts all M-type tags in an HTML page, as well as the text within each M-type tag. The text includes, for example, "Read Now," "All Chapters," and "Chapter 1." Based on the text within these M-type tags, it can be predicted that the target website corresponding to the HTML page is a novel website.

[0130] In an embodiment of the present application, since the text information in HTML tags is also different in different types of websites, when identifying the website type of the target website visited by the terminal device based on the text information in the HTML tags associated with the specified website type, the accuracy of website type identification can be effectively improved, thereby improving the accuracy of the effectiveness of the detection and extraction rules.

[0131] Step 403: When the preset rule library includes a first extraction rule that matches the website link, the server extracts key information from the web page data using the first extraction rule to obtain target key information.

[0132] Specifically, after receiving web page data, the server first formats the web page data, removing redundant data (such as advertising tags). It then searches a preset rule library using the website link as a keyword, where the website link can be the website's domain name. If the preset rule library contains a first extraction rule that matches the website link, the rule parsing engine uses the first extraction rule to extract key information from the web page data, obtaining the target key information.

[0133] In some embodiments, the key information in webpage data may include multiple types of business content. For example, a novel website may include different types of business content, such as the book title, author name, table of contents, completion status, book type, and book text. Therefore, an extraction rule may include multiple sub-rules, each corresponding to extracting a type of business content.

[0134] For example, an extraction rule includes 5 sub-rules, namely sub-rule 1 to sub-rule 5, among which sub-rule 1 is used to extract the "book name" from the novel website; sub-rule 2 is used to extract the "author name" from the novel website; sub-rule 3 is used to extract the "target list" from the novel website; sub-rule 4 is used to extract the "completion status" from the novel website; and sub-rule 5 is used to extract the "book text" from the novel website.

[0135] In some embodiments, if the preset rule library does not include the first extraction rule that matches the website link, the cache area is queried using the website link as a keyword to obtain the number of failed matches for the website link;

[0136] When the number of failed matches of the website link is greater than a preset threshold, a second extraction rule corresponding to the target website is predicted based on the characteristics of the key information contained in the web page data, and the second extraction rule is added to the preset rule library.

[0137] Specifically, the cache can be a Redis database, a Memcached system, an Ehcache database, or the like. The cache records the number of times each website link fails to match the extraction rules. If the preset rule library does not contain the first extraction rule that matches the website link, it indicates that the extraction rule has failed to match. The cache is then searched using the website link as a keyword to obtain the number of failed matches for the website link.

[0138] An upper limit for the number of matching failures is preset. When the number of matching failures does not reach the upper limit, that is, when the number of matching failures of the website link is not greater than the preset threshold, the number of matching failures of the website link is increased by one to obtain an updated number of matching failures of the website link.

[0139] When the number of failed matches reaches the upper limit, it indicates that there are many users accessing the website link. Therefore, it is necessary to filter interference factors from the web page data of the corresponding website to improve the user experience of the website. Based on this, when the number of failed matches for the website link exceeds the preset threshold, the application uses the trained large language model to predict the second extraction rule corresponding to the target website based on the characteristics of the key information contained in the web page data, and adds the second extraction rule to the preset rule library.

[0140] In the specific implementation, a pre-training-fine-tuning training method is used to train a large language model, and then the trained large language model is used to predict the extraction rules. Specifically, it includes the sample preparation stage, the training stage, and the application stage. Figure 6 These stages are described in detail.

[0141] In the sample preparation stage: historical rule parsing data is obtained, and then the historical rule parsing data is cleaned and split to obtain multiple valid rule parsing data, where each valid rule parsing data includes: website links, HTML pages, extraction rules and rule parsing results. The rule parsing results refer to: key information extracted from the HTML page using the extraction rules.

[0142] The HTML pages and extracted rules from the valid rule parsing data are combined to form a mapping pair. The mapping pairs are then labeled based on the correctness of the rule parsing results in the valid rule parsing data. Specifically, if the rule parsing result is incorrect, the mapping pair is labeled as a negative sample; if the rule parsing result is correct, the mapping pair is labeled as a positive sample. The labeled positive and negative samples form a sample set, where the number of positive samples in the sample set is much larger than the number of negative samples. Training and test corpora are constructed based on the sample set.

[0143] During the training phase, the training corpus is fed into a pre-trained large language model (e.g., ChatGLM-6B) for fine-tuning, yielding a large language model for rule prediction. The large language model is then tested for accuracy using the test corpus. If the accuracy passes, the trained large language model is output.

[0144] In the application stage: the web page data is input into the trained large language model, the trained large language model extracts features from the web page data, obtains the feature vector of the key information contained in the web page data, and then predicts the second extraction rule corresponding to the target website based on the feature vector of the key information, and then adds the second extraction rule to the preset rule library.

[0145] In an embodiment of the present application, a cache area is set to record the number of matching failures of each website link matching extraction rule. When the number of matching failures is not greater than a preset threshold, the number of matching failures is updated. When the number of matching failures is greater than the preset threshold, based on the characteristics of the key information contained in the web page data, the second extraction rule corresponding to the target website is predicted, and the second extraction rule is added to the preset rule library, avoiding the resource overhead caused by frequent prediction of extraction rules, while ensuring the maintenance of the extraction rules of hot websites, thereby improving the effect of filtering interference factors of hot websites.

[0146] In step 404 , the server performs format verification and content verification on the target key information, and if at least one of the format verification and content verification fails, the server marks the first extraction rule as invalid.

[0147] Specifically, when the target key information includes business content extracted by multiple sub-rules, format verification and content verification can be performed on each of the extracted business content to obtain verification results for each business content. If the verification result for at least one business content fails, the first extraction rule can be marked invalid, and the corresponding sub-rule within the first extraction rule can also be marked invalid.

[0148] Of course, you can also merge multiple text contents to perform format verification and content verification to obtain the verification result of the target key information. When the verification result is not passed, the first extraction rule is marked as invalid. This application does not make specific restrictions on this.

[0149] For example, see Figure 7 When the user browses the novel web page B in the novel website A through the browser application, the terminal device identifies the type of the currently visited website as a novel website through the site, and uploads the HTML page 701 of the novel web page B to the server.

[0150] The server retrieves book source rule C from the book source rule library, which matches the website link of novel website A. Book source rule C includes sub-rules C1, C2, C3, and C4. Sub-rule C1 extracts the book title, C2 extracts the author name, C3 extracts the completion status, and C4 extracts the book genre. The status of book source rule C, sub-rules C1, C2, C3, and C4 are all valid.

[0151] Assume that the book name extracted from the HTML page using sub-rule c1 is: XXX, the author name extracted from the HTML page using sub-rule c2 is: abc, the completion status extracted from the HTML page using sub-rule c3 is: martial arts novel, and the book type extracted from the HTML page using sub-rule c4 is: completed.

[0152] The format check and content check are performed on the extracted book name "XXX". If the check result is passed, the status of sub-rule c1 in the book source rule C remains: valid.

[0153] The format check and content check are performed on the extracted author name "abc". If the check result is passed, the status of sub-rule c2 in the book source rule C remains: valid.

[0154] The format check and content check are performed on the extracted completion status "Martial Arts Novels". If the check result is failure, the status of sub-rule c3 in the book source rule C is marked as invalid.

[0155] The format check and content check are performed on the extracted book type "Completed". If the check result is failed, the status of sub-rule c4 in the book source rule C is marked as invalid.

[0156] Finally, the source rule C is marked as invalid.

[0157] In an embodiment of the present application, after the terminal device requests to access the target website, it obtains the rendered web page data and uploads the web page data to the server. That is, multiple terminal devices access the target website in a distributed manner and parse to obtain the web page data. Compared with the technical solution in which a single-point server periodically polls to obtain the source code of the website from the website server and parses the source code into web page data, the present application avoids the problem of the website server banning the IP address of the single-point server, and ensures the correctness of the obtained web page data.

[0158] Secondly, in the client-side rendering mode, the rendered web page data uploaded by the terminal device is complete web page data. Therefore, there is no need to run an additional headless browser on the server to render the original page data to obtain complete web page data, and then perform validity detection of the extraction rules. This not only reduces the memory consumption of the server, but also improves the efficiency of the validity detection of the extraction rules.

[0159] In addition, the server obtains matching extraction rules based on the website links contained in the web page data, and then uses the extraction rules to extract key information from the web page data, and verifies the correctness of the key information to obtain the validity test results of the extraction rules, thereby realizing real-time and rapid detection of invalid extraction rules in the preset rule library, so that the invalid extraction rules can be maintained in a timely manner in the future, ensuring the effect of filtering interference factors in the web page data and improving the user experience of the website.

[0160] In some embodiments, related technologies, when performing rule validity checks, are limited in their verification of the correctness of key information, typically assuming that the key information contains specified content. In this case, even when the extraction results contain not only the specified content but also other content, the extraction rule is still deemed valid. For example, if the key information extracted using a book source rule includes not only the book title but also meaningless special characters, then, because the key information contains the specified content "book title," the book source rule is directly deemed valid. This determination is essentially crude and has low accuracy.

[0161] In view of this, in the embodiment of the present application, the target key information is formatted based on the preset text format rules. If the format verification passes, the text type corresponding to the target key information is predicted based on the text semantic features of the target key information, and the content of the target key information is verified based on the text type.

[0162] Specifically, the text formatting rules can be set according to actual conditions. For example, the key information cannot be empty, the key information cannot contain special symbols, and the length of the key information cannot exceed a preset length.

[0163] During content verification, a pre-trained text classification model is used to classify correct and incorrect text content. In practice, if key information contains other content besides the specified content, the text classification model will determine that the key information is incorrect.

[0164] At the end of training, the target key information is input into the text classification model. The model then extracts features from the target key information, obtaining semantic features. Based on these features, the text type corresponding to the target key information is predicted. If the text type is incorrect, the content verification fails; if the text type is correct, the content verification passes.

[0165] In actual application, the target key information may be format-checked first. If the format check passes, the content check is then performed. If the content check passes, the first extraction rule remains valid. If the content check fails, the first extraction rule is marked as invalid.

[0166] If the format check fails, the first extraction rule is directly marked as invalid.

[0167] In addition to the aforementioned implementation of first performing format verification on the target key information and then performing content verification, the target key information can also be verified for content first and then for format verification. Of course, format verification and content verification can also be performed on the target key information in parallel, and this application does not make specific limitations on this.

[0168] In an embodiment of the present application, the format of key information is checked based on preset text format rules, and the content of key information is checked using a text classification model. Through a multi-round verification method that combines format verification and content verification, the accuracy of key information verification is greatly improved, thereby improving the accuracy of the effectiveness of detection and extraction rules.

[0169] In some embodiments, after the first extraction rule is marked as invalid, a second extraction rule corresponding to the target website is predicted based on features of key information contained in the web page data, and the second extraction rule is added to the preset rule library.

[0170] Specifically, the trained large language model is used to predict the second extraction rule corresponding to the target website based on the characteristics of the key information contained in the webpage data. This second extraction rule is then added to the preset rule library. In the specific implementation, the large language model is trained using a pre-training and fine-tuning training method. The training process of the large language model has been described above and will not be repeated here.

[0171] For example, see Figure 8 ,Suppose that when the user browses the novel webpage B in the novel website A through a browser, the book source rule C is used to extract the key information in the HTML page corresponding to the novel webpage B, and after performing format verification and content verification, the book source rule C is marked as invalid.

[0172] At this point, the server inputs the HTML page of novel website B into the trained large language model and obtains source rule D. Source rule D includes sub-rules d1, d2, d3, and d4. All of these sub-rules are in the valid state. Source rule D is then associated with the website link of novel website A and stored in the source rule library.

[0173] In an embodiment of the present application, after the first extraction rule is marked as invalid, the second extraction rule corresponding to the target website is predicted based on the characteristics of the key information contained in the web page data, and then the second extraction rule is added to the preset rule library. In this way, when it is detected that the extraction rule is invalid, the invalid extraction rule is supplemented and maintained in time, thereby ensuring that the extraction rules in the rule library continue to be effective, thereby improving the effect of filtering interference factors in web page data.

[0174] In some embodiments, when some sub-rules in the first extraction rule are marked as invalid, while other sub-rules are still marked as valid, for each invalid sub-rule, the HTML code snippet corresponding to the sub-rule is obtained from the webpage data, and then the HTML code snippet is input into the trained large language model for prediction to obtain the corresponding new sub-rule. Different large language models can be used for prediction for different sub-rules.

[0175] Sub-rules marked as invalid in the first extraction rule are replaced with new sub-rules obtained by prediction, and the new sub-rules are marked as valid. After all sub-rules marked as invalid in the first extraction rule are replaced with new sub-rules and marked as valid, a second extraction rule is obtained.

[0176] When the terminal device subsequently accesses a web page in the target website, the server obtains the matching second extraction rule from the rule library, and then uses the second extraction rule to extract key information from the web page data, and returns the key information to the terminal device. The terminal device displays the key information in the traffic meter application.

[0177] For example, see Figure 9 , assuming that the user browses the novel webpage B in the novel website A through a browser, and uses the book source rule C to extract the key information in the HTML page corresponding to the novel webpage B, and after performing format verification and content verification, the book source rule C is marked as invalid, and the status of sub-rule c1 is: valid; the status of sub-rule c2 remains: valid; the status of sub-rule c3 is: invalid; the status of sub-rule c4 is: invalid.

[0178] The server inputs the HTML code snippet corresponding to sub-rule c3 in the HTML page into the trained large language model to obtain a new sub-rule c5; and inputs the HTML code snippet corresponding to sub-rule c4 in the HTML page into the trained large language model to obtain a new sub-rule c6.

[0179] Replace sub-rule c3 in source rule C with new sub-rule c5 and mark new sub-rule c5 as valid. Replace sub-rule c4 in source rule C with new sub-rule c6 and mark new sub-rule c6 as valid. Mark source rule C as valid.

[0180] In order to better explain the embodiment of the present application, the following introduces a method for detecting the validity of an extraction rule provided by the embodiment of the present application in combination with a novel website as a specific implementation scenario. The process of the method can be interactively executed by a terminal device and a server, and includes the following steps: Figure 10 As shown:

[0181] Step 1001: A terminal device accesses a web page in a target website through a browser application.

[0182] Step 1002: The terminal device identifies the website type of the target website through a site identification script.

[0183] In step 1003 , the terminal device determines whether the website type is a novel website. If so, step 1004 is executed; otherwise, step 1018 is executed.

[0184] Step 1004: The terminal device uploads the HTML page of the web page to the cloud server.

[0185] Step 1005: The cloud server cleans the HTML page.

[0186] Step 1006: The cloud server queries the book source rule library based on the website link included in the HTML page.

[0187] Step 1007 , determining whether the book source rule library contains a matching book source rule; if so, executing step 1008 ; otherwise, executing step 1015 .

[0188] In step 1008, the cloud server parses the HTML page using the matching book source rules to obtain basic information about the book.

[0189] Step 1009: The cloud server performs format verification and content verification on the basic information of the book.

[0190] Step 1010, determine whether all verifications are passed, if so, execute step 1011, otherwise execute step 1012.

[0191] Step 1011: The cloud server keeps the status of the book source rule valid.

[0192] Step 1012: The cloud server marks the book source rule as invalid.

[0193] In step 1013, the cloud server inputs the HTML page into the large language model to obtain new book source rules.

[0194] In step 1014, the cloud server associates the new book source rule with the website link and stores it in the book source rule library.

[0195] In step 1015, the cloud server queries Redis based on the website link to obtain the corresponding number of matching failures.

[0196] Step 1016, determine whether the number of matching failures is greater than 1000 times, if so, execute steps 1013 and 1014, otherwise execute step 1017.

[0197] Step 1017: Increase the number of matching failures by one.

[0198] Step 1018, exit the process.

[0199] In the embodiment of the present application, after the terminal device requests to access the target website, it obtains the rendered web page data and uploads the web page data to the server. That is, multiple terminal devices access the target website in a distributed manner and parse and obtain the web page data. Compared with the technical solution in which a single-point server periodically polls to obtain the website source code from the website server and parses the source code into web page data, the present application avoids the problem of the website server blocking the IP address of the single-point server and ensures the accuracy of the obtained web page data.

[0200] Secondly, in the client-side rendering mode, the rendered web page data uploaded by the terminal device is complete web page data. Therefore, there is no need to run an additional headless browser on the server to render the original page data to obtain complete web page data, and then perform validity detection of the extraction rules. This not only reduces the memory consumption of the server, but also improves the efficiency of the validity detection of the extraction rules.

[0201] In addition, the server obtains matching extraction rules based on the website links contained in the web page data, and then uses the extraction rules to extract key information from the web page data, and verifies the correctness of the key information to obtain the validity test results of the extraction rules, thereby realizing real-time and rapid detection of invalid extraction rules in the preset rule library, so that the invalid extraction rules can be maintained in a timely manner in the future, ensuring the effect of filtering interference factors in the web page data and improving the user experience of the website.

[0202] Based on the same technical concept, the embodiment of the present application provides a structural diagram of a device for detecting the validity of an extraction rule, which is applied to a server, such as Figure 11 As shown, the validity detection device 1100 of the extraction rule includes:

[0203] The receiving module 1101 is configured to receive rendered web page data sent by a terminal device, the web page data being obtained after the terminal device requests access to a target website, and the web page data at least including a website link of the target website;

[0204] An extraction module 1102 is configured to extract key information from web page data using the first extraction rule to obtain target key information when the preset rule library contains a first extraction rule that matches the website link;

[0205] The verification module 1103 is configured to perform format verification and content verification on the target key information, and mark the first extraction rule as invalid if at least one of the format verification and content verification fails.

[0206] Optionally, an update module 1104 is also included;

[0207] The update module 1104 is specifically used for:

[0208] After marking the first extraction rule as invalid, predicting a second extraction rule corresponding to the target website based on the characteristics of key information contained in the web page data;

[0209] Add the second extraction rule to the preset rule library.

[0210] Optionally, the extraction module 1102 is further configured to:

[0211] If the preset rule library does not contain the first extraction rule that matches the website link, the cache area is searched using the website link as a keyword to obtain the number of failed matches of the website link;

[0212] When the number of failed matches of the website link is greater than a preset threshold, a second extraction rule corresponding to the target website is predicted based on the characteristics of the key information contained in the web page data, and the second extraction rule is added to the preset rule library.

[0213] Optionally, the extraction module 1102 is further configured to:

[0214] If the number of failed matches of the website link is not greater than the preset threshold, the number of failed matches of the website link is increased by one to obtain an updated number of failed matches of the website link.

[0215] Optionally, the verification module 1103 is specifically configured to:

[0216] Perform format verification on target key information based on preset text format rules;

[0217] When the format check passes, the text type corresponding to the target key information is predicted based on the text semantic features of the target key information, and the content of the target key information is checked based on the text type.

[0218] Based on the same technical concept, the embodiment of the present application provides a schematic diagram of a structure of a device for detecting the validity of an extraction rule, which is applied to a terminal device, such as Figure 12 As shown, the validity detection device 1200 of the extraction rule includes:

[0219] The acquisition module 1201 is used to request access to a target website and obtain rendered web page data corresponding to the target website, where the web page data at least includes a website link of the target website;

[0220] The sending module 1202 is used to send the web page data to the server so that when the server determines that the preset rule library contains a first extraction rule that matches the website link, it uses the first extraction rule to extract key information from the web page data to obtain the target key information; and performs format verification and content verification on the target key information, and when at least one of the format verification and content verification fails, the first extraction rule is marked as invalid.

[0221] Optionally, it further includes an identification module 1203;

[0222] The identification module 1203 is specifically used for:

[0223] Before sending the web page data to the server, the website type of the target website is identified based on the web page data through the site identification script, and the website type is confirmed to be the specified website type.

[0224] Optionally, the identification module 1203 is specifically configured to:

[0225] Extract HTML tags associated with a specific website type from web page data using a site identification script;

[0226] Identify the website type of the target website based on the text information within the extracted HTML tags.

[0227] Optionally, the web page data is rendered into an HTML page using a client-side rendering mode or a server-side rendering mode.

[0228] In an embodiment of the present application, after the terminal device requests to access the target website, it obtains the rendered web page data and uploads the web page data to the server. That is, multiple terminal devices access the target website in a distributed manner and parse to obtain the web page data. Compared with the technical solution in which a single-point server periodically polls to obtain the source code of the website from the website server and parses the source code into web page data, the present application avoids the problem of the website server banning the IP address of the single-point server, and ensures the correctness of the obtained web page data.

[0229] Secondly, in the client-side rendering mode, the rendered web page data uploaded by the terminal device is complete web page data. Therefore, there is no need to run an additional headless browser on the server to render the original page data to obtain complete web page data, and then perform validity detection of the extraction rules. This not only reduces the memory consumption of the server, but also improves the efficiency of the validity detection of the extraction rules.

[0230] In addition, the server obtains matching extraction rules based on the website links contained in the web page data, and then uses the extraction rules to extract key information from the web page data, and verifies the correctness of the key information to obtain the validity test results of the extraction rules, thereby realizing real-time and rapid detection of invalid extraction rules in the preset rule library, so that the invalid extraction rules can be maintained in a timely manner in the future, ensuring the effect of filtering interference factors in the web page data and improving the user experience of the website.

[0231] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1 The terminal device and / or server shown, such as Figure 13 As shown, it includes at least one processor 1301 and a memory 1302 connected to the at least one processor. The specific connection medium between the processor 1301 and the memory 1302 is not limited in the embodiment of the present application. Figure 13 For example, the processor 1301 and the memory 1302 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0232] In an embodiment of the present application, the memory 1302 stores instructions that can be executed by at least one processor 1301. The at least one processor 1301 can execute the steps of the above-mentioned method for detecting the validity of the extraction rules by executing the instructions stored in the memory 1302.

[0233] The processor 1301 is the control center of the computer device and can connect various parts of the computer device using various interfaces and lines. It can run or execute instructions stored in the memory 1302 and call data stored in the memory 1302 to implement the validity detection of the extraction rules. Optionally, the processor 1301 may include one or more processing units. The processor 1301 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understood that the modem processor may not be integrated into the processor 1301. In some embodiments, the processor 1301 and the memory 1302 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0234] The processor 1301 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0235] Memory 1302 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1302 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RandomAccess Memory, RAM), a static random access memory (Static RandomAccess Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Memory 1302 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 1302 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0236] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned extraction rule validity detection method.

[0237] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned extraction rule validity detection method.

[0238] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0239] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0240] These computer program instructions may also be stored in a computer readable memory that can direct a computer device or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0241] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operating steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0242] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0243] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for detecting the validity of an extraction rule, characterized in that: Applicable to servers, including: Receiving rendered web page data sent by a terminal device, wherein the web page data is obtained after the terminal device requests to access a target website, and the web page data at least includes a website link of the target website; When the preset rule library contains a first extraction rule that matches the website link, extract key information from the web page data using the first extraction rule to obtain target key information; A format check and a content check are performed on the target key information, and when at least one of the format check and the content check fails, the first extraction rule is marked as invalid.

2. The method according to claim 1, wherein After marking the first extraction rule as invalid, the method further includes: Predicting a second extraction rule corresponding to the target website based on features of key information contained in the webpage data; Add the second extraction rule to the preset rule library.

3. The method according to claim 1, wherein Also includes: If the preset rule library does not include the first extraction rule that matches the website link, querying the cache area using the website link as a keyword to obtain the number of failed matches for the website link; When the number of failed matches of the website link is greater than a preset threshold, a second extraction rule corresponding to the target website is predicted based on features of key information contained in the web page data, and the second extraction rule is added to the preset rule library.

4. The method according to claim 3, wherein Also includes: If the number of failed matches of the website link is not greater than the preset threshold, the number of failed matches of the website link is increased by one to obtain an updated number of failed matches of the website link.

5. The method according to any one of claims 1 to 4, characterized in that: The format verification and content verification of the target key information includes: Performing format verification on the target key information based on preset text format rules; When the format check passes, the text type corresponding to the target key information is predicted based on the text semantic features of the target key information, and the content check of the target key information is performed based on the text type.

6. A method for detecting the validity of an extraction rule, characterized in that: Applied to terminal equipment, including: Requesting access to a target website and obtaining rendered web page data corresponding to the target website, wherein the web page data at least includes a website link to the target website; The web page data is sent to a server, so that when the server determines that a preset rule library contains a first extraction rule that matches the website link, the server uses the first extraction rule to extract key information from the web page data to obtain target key information; and performs format verification and content verification on the target key information, and when at least one of the format verification and the content verification fails, the first extraction rule is marked as invalid.

7. The method according to claim 6, wherein Before sending the web page data to the server, the method further includes: The website type of the target website is identified based on the web page data through a site identification script, and the website type is confirmed to be a specified website type.

8. The method according to claim 7, wherein The identifying the website type of the target website based on the web page data by using a site identification script includes: extracting HTML tags associated with the specified website type from the web page data using the site identification script; Based on the text information extracted from the HTML tags, the website type of the target website is identified.

9. The method according to any one of claims 6 to 8, characterized in that The web page data is an HTML page rendered in a client-side rendering mode or a server-side rendering mode.

10. A device for detecting the validity of extraction rules, characterized in that: Applicable to servers, including: a receiving module, configured to receive rendered web page data sent by a terminal device, wherein the web page data is obtained after the terminal device requests to access a target website, and the web page data at least includes a website link of the target website; an extraction module configured to extract key information from the web page data using a first extraction rule matching the website link when a preset rule library contains the first extraction rule to obtain target key information; The verification module is configured to perform format verification and content verification on the target key information, and mark the first extraction rule as invalid when at least one of the format verification and the content verification fails.

11. A device for detecting the validity of an extraction rule, characterized in that: Applied to terminal equipment, including: An acquisition module, configured to request access to a target website and obtain rendered web page data corresponding to the target website, wherein the web page data at least includes a website link of the target website; A sending module is used to send the web page data to a server so that when the server determines that a preset rule library contains a first extraction rule that matches the website link, it uses the first extraction rule to extract key information from the web page data to obtain target key information; and performs format verification and content verification on the target key information, and when at least one of the format verification and the content verification fails, marks the first extraction rule as invalid.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of any one of the methods according to claims 1 to 9.

14. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer device, the computer device is caused to perform the steps of the method according to any one of claims 1 to 9.

Images