Information extraction method and page structured extraction method

By filtering and using the target code to extract information on the target page, the problem of low efficiency of web page information extraction in the existing technology is solved, and efficient automatic information extraction is achieved.

CN118981564APending Publication Date: 2024-11-19ALIBABA (CHINA) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410833083.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the prior art, web page information extraction efficiency is low, and an efficient information extraction solution is needed to cope with the explosive growth of web page data.

Method used

By determining the page information of the target page, including page type and routing information, the target code is filtered out from the candidate code, and the target code is used to extract information on the target page to achieve automated information extraction.

Benefits of technology

It improves the efficiency of information extraction, realizes automatic information extraction based on target code, and reduces the need for manual development scripts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981564A_ABST
    Figure CN118981564A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an information extraction method and a page structured extraction method.The information extraction method comprises the steps that in response to an information extraction request for a target page, page information of the target page is determined, and the page information comprises a page type and routing information; according to the page type and the routing information, target codes are screened out from candidate codes, and the candidate codes are obtained by conducting code writing on the basis of reference page information of the reference page; and performing information extraction on the target page by utilizing the target code to obtain an information extraction result of the target page. According to the information extraction method, the target code is screened out from the candidate codes, and the target code is utilized to perform information extraction on the target page, so that automatic information extraction based on the target code is realized, and the information extraction efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular to information extraction methods and page structured extraction methods. Background Art

[0002] With the development of computer technology, web page data has exploded, and web information extraction has gradually become a research focus. Web information extraction refers to the process of extracting valuable or target data from Hypertext Markup Language (HTML) web pages.

[0003] At present, due to the diversity of web pages, scripts are usually developed manually, and the developed scripts are used to extract web page information. However, manual script development takes a lot of time, resulting in extremely low efficiency in web page information extraction. Therefore, an efficient information extraction solution is urgently needed. Summary of the invention

[0004] In view of this, an embodiment of this specification provides an information extraction method. One or more embodiments of this specification also relate to a page structure extraction method, an information extraction device, a page structure extraction device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an information extraction method, comprising:

[0006] In response to an information extraction request for a target page, determining page information of the target page, wherein the page information includes a page type and routing information;

[0007] Filtering target codes from candidate codes according to page type and routing information, wherein the candidate codes are obtained by coding based on reference page information of reference pages;

[0008] The target code is used to extract information from the target page to obtain the information extraction result of the target page.

[0009] According to a second aspect of an embodiment of this specification, a page structure extraction method is provided, comprising:

[0010] In response to a page structure extraction request for a target page, determining page information of the target page, wherein the page information includes a page type and routing information;

[0011] Filtering target codes from candidate codes according to page type and routing information, wherein the candidate codes are obtained by coding based on reference page information of reference pages;

[0012] The target code is used to extract structured information of the target page to obtain a page structure extraction result of the target page.

[0013] According to a third aspect of the embodiments of this specification, there is provided an information extraction device, including:

[0014] A first determination module is configured to determine page information of a target page in response to an information extraction request for the target page, wherein the page information includes a page type and routing information;

[0015] A first screening module is configured to screen target codes from candidate codes according to page type and routing information, wherein the candidate codes are obtained by coding based on reference page information of a reference page;

[0016] The first extraction module is configured to extract information from a target page using the target code to obtain an information extraction result of the target page.

[0017] According to a fourth aspect of the embodiments of this specification, a page structure extraction device is provided, including:

[0018] A second determination module is configured to determine page information of the target page in response to a page structure extraction request for the target page, wherein the page information includes a page type and routing information;

[0019] A second screening module is configured to screen target codes from candidate codes according to page type and routing information, wherein the candidate codes are obtained by coding based on reference page information of the reference page;

[0020] The second extraction module is configured to extract structured information from the target page using the target code to obtain a page structure extraction result of the target page.

[0021] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0022] Memory and processor;

[0023] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method provided in the first aspect or the second aspect are implemented.

[0024] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the method provided in the first aspect or the second aspect are implemented.

[0025] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in the first aspect or the second aspect above.

[0026] An information extraction method provided by an embodiment of the present specification includes: in response to an information extraction request for a target page, determining page information of the target page, wherein the page information includes page type and routing information; filtering out a target code from candidate codes according to the page type and routing information, wherein the candidate code is obtained by coding based on reference page information of a reference page; using the target code, extracting information from the target page to obtain an information extraction result of the target page. By filtering out the target code from the candidate codes and extracting information from the target page using the target code, automatic information extraction based on the target code is achieved, thereby improving information extraction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is an architecture diagram of an information extraction system provided by an embodiment of this specification;

[0028] Figure 2 is an architecture diagram of another information extraction system provided by an embodiment of this specification;

[0029] Figure 3 is a flow chart of an information extraction method provided by an embodiment of this specification;

[0030] Figure 4 is a flow chart of a page structure extraction method provided by an embodiment of this specification;

[0031] Figure 5 is a processing flow chart of an information extraction method provided by an embodiment of this specification;

[0032] Figure 6 is a schematic diagram of the structure of an information extraction device provided by an embodiment of this specification;

[0033] Figure 7 It is a schematic diagram of the structure of a page structure extraction device provided by an embodiment of this specification;

[0034] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0035] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0036] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0038] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model (Foundation Model 1). The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model 1), a multi-modal pre-training model (multi-modal pre-training model 1), etc.

[0040] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0041] First, the terms involved in one or more embodiments of this specification are explained.

[0042] Large-scale language model: A type of model trained using deep learning technology, designed for a variety of language processing tasks such as understanding, generating, translating or summarizing text. Large-scale language models can capture and simulate the complexity of human language. They are characterized by being pre-trained on a large text library and then adapted to specific tasks or fields through a small amount of targeted training (fine-tuning).

[0043] Visual-Language Model (VLM): A multimodal model that combines the processing power of vision (images, videos) and text to understand and generate information across these modalities. For example, VLM can be used for image annotation (generating descriptive text based on the content of an image), visual question answering (answering questions about a given image), or text-to-image generation (generating relevant images based on a text description). VLM further expands machine understanding of complex human communication by fusing technologies from the fields of computer vision and natural language processing.

[0044] Agent: In artificial intelligence and computer science, an agent usually refers to a software entity that can automatically perform tasks. These tasks can be simple tasks (such as automatically responding to emails) or complex tasks (such as the control system of a self-driving car). An agent can perceive its environment (through sensors or data input), make decisions (through some form of data processing or learning algorithm) and perform actions (through activators or output data). Intelligent agents may include search engines, recommendation systems, automated trading systems, etc.

[0045] Page structured extraction: refers to the process of automatically extracting structured data from HTML web pages.

[0046] Code Interpreter: A software tool suitable for rapid prototyping and script execution. It reads, analyzes and executes program code line by line in sequence. Unlike a compiler, a code interpreter does not need to convert the entire program code into machine language code before executing the program, but interprets and executes it on the fly. This means that the code can be run immediately, making it easier to develop and debug programs. Languages ​​such as Python, Ruby and Hypertext Preprocessor (PHP) often use code interpreters to execute code.

[0047] In this specification, an information extraction method is provided. This specification also involves a page structured extraction method, an information extraction device, a page structured extraction device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0048] See also Figure 1 , Figure 1 An architecture diagram of an information extraction system provided by an embodiment of the present specification is shown. The information extraction system may include a client 100 and a server 200;

[0049] The client 100 is used to send an information extraction request for a target page to the server 200;

[0050] The server 200 is used to respond to the information extraction request for the target page, determine the page information of the target page, wherein the page information includes the page type and routing information; select the target code from the candidate codes according to the page type and the routing information, wherein the candidate code is obtained by coding based on the reference page information of the reference page; extract information from the target page using the target code to obtain the information extraction result of the target page; and send the information extraction result of the target page to the client 100;

[0051] The client 100 is also used to receive the information extraction result of the target page sent by the server 200.

[0052] By applying the solution of the embodiment of this specification, the target code is screened out from the candidate codes, and the target code is used to extract information from the target page, thereby realizing automatic information extraction based on the target code and improving the efficiency of information extraction.

[0053] See also Figure 2 , Figure 2 The architecture diagram of another information extraction system provided by an embodiment of the present specification is shown, and the information extraction system may include multiple clients 100 and a server 200. Multiple clients 100 may establish a communication connection through the server 200. In the information extraction scenario, the server 200 is used to provide information extraction services between multiple clients 100. Multiple clients 100 may respectively serve as a sender or a receiver, and realize communication through the server 200.

[0054] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the information extraction scenario, the user can issue an information extraction request to the server 200 through the client 100, and the server 200 generates an information extraction result according to the information extraction request, and pushes the information extraction result to other clients that have established communication.

[0055] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0056] The client 100 can be a browser, an application (APP, App l icat ion), or a web application such as HyperText Markup Language Version 5 (H5, HyperText Markup Language 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device, and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, mailbox clients, social platform software, etc.

[0057] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that provide support for models used on the client, and servers that process data sent by the client. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content De l ivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0058] It is worth noting that the information extraction method provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the information extraction method provided in the embodiments of this specification. In other embodiments, the information extraction method provided in the embodiments of this specification may also be jointly executed by the client and the server. Next, taking the server executing the information extraction method proposed in the embodiments of this specification as an example, the processing flow of the information extraction method is described.

[0059] See also Figure 3 , Figure 3A flowchart of an information extraction method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0060] Step 302: In response to an information extraction request for a target page, determine page information of the target page, wherein the page information includes a page type and routing information.

[0061] In one or more embodiments of the present specification, the client may send an information extraction request for a target page to the server to request the server to extract information from the target page. After receiving the information extraction request sent by the client, the server may determine the page information of the target page, and then use the page information to extract information from the target page to obtain an information extraction result for the target page.

[0062] Specifically, the target page can be different types of web pages, such as pages of news websites, pages of social media platforms, pages of forums and communities, pages of corporate official websites, etc. Information extraction requests can be requests in different scenarios, such as news aggregation scenarios, e-commerce competitive intelligence acquisition scenarios, social media data crawling scenarios, etc.

[0063] Page information includes page type, routing information and page elements. Page type refers to the classification of web pages according to their functions, layout or content characteristics. This classification helps developers and analysis tools to take different approaches to different types of pages. For example, a website may contain multiple page types, such as home page, product detail page, article page, list page, etc. Routing information can be called Uniform Resource Locator (URL) path or routing path. Routing information is the part of the web page address (URL) that indicates the location of a specific resource or page. It usually follows the domain name and is defined by a path separator (such as " / "). Page elements are the basic components of a page, including but not limited to text content, images, hyperlinks, forms, videos, audios, scripts and style sheets. In HTML, page elements are defined and organized by tags, attributes, and classes.

[0064] Information extraction requests can request the server to extract different types of information from the target page, such as structured information and unstructured information. The specific type of information to be extracted depends on the purpose of the request and the content of the target page. For example, in an e-commerce scenario, the focus may be on product names, prices, and ratings; while in a news site, the focus may be more on titles, authors, publishing times, and article content. Structured information refers to highly organized and standardized data that can be presented in a table format, with each column representing a data field and each row representing a record. Structured information includes but is not limited to text content (such as article body, product description, news headline, comment text, etc.), numbers and quantities (such as prices, ratings, statistics, quantities, percentages, etc.), dates and times (such as publishing dates, update times, event start / end times, etc.), lists and tabular data (such as product lists, stock quotes, weather forecasts, sports scores, etc.). Unstructured information is data that has no predefined format or organizational structure, such as images, videos, audio, etc. Semi-structured information is between structured and unstructured. It does not fully conform to a fixed data model, but contains some tags or labels that can be used to identify and separate data elements. This type of information usually exists in formats such as eXtensible Markup Language (XML) and JSON (JavaScript Object Notation). Although there is no fixed column and row structure, there is a certain hierarchy or tag structure inside, so that the data part can be parsed and extracted.

[0065] In actual applications, there are many ways to determine the page information of the target page, which can be selected according to the actual situation. The embodiments of this specification do not impose any restrictions on this. For the routing information of the target page, in one possible implementation, the JavaScript Location object can be used to determine the routing information of the target page. In another possible implementation, the routing interface (API, Application Programming Interface) of the front-end framework can be used to determine the routing information of the target page. For the page type of the target page, in one possible implementation, the page tag (such as class name, data tag, etc.) can be found from the page source code, and the page type can be determined based on the page tag. In another possible implementation, the page type can be determined by analyzing the path information. For example, the presence of / product / in the URL may indicate that this is a product details page, while / news / may correspond to a news or article page.

[0066] Step 304: Filter out target codes from candidate codes according to the page type and routing information, wherein the candidate codes are compiled based on reference page information of the reference page.

[0067] In one or more embodiments of the present specification, in response to an information extraction request for a target page, after determining the page information of the target page, further, a target code that can extract information from the target page can be screened out from pre-generated candidate codes.

[0068] Specifically, the number of candidate codes can be one or more. The candidate code here can be a candidate code that has passed the test or an untested candidate code. The target code refers to a candidate code whose reference routing information and reference page type match the routing information and page type of the target page. The reference routing information and reference page type corresponding to the candidate code refer to the reference routing information and reference page type of the reference page used to write the candidate code. In practical applications, when the target code is screened out from the candidate code according to the page type and routing information, the target code can be screened out from multiple candidate codes that have passed the test according to the page type and routing information. Specifically, in a possible implementation of this specification, the page type and routing information can be converted into a feature vector that can be processed by a machine learning model, and the feature vector is input into a classification model. The classification model can perform vector matching on the candidate feature vectors of multiple candidate codes that have passed the test and the input feature vector, and recommend the target code from multiple candidate codes that have passed the test according to the vector matching result. Among them, the vector matching method includes but is not limited to cosine similarity calculation and Euclidean distance calculation, which are selected according to the actual situation, and the embodiment of this specification does not make any limitation on this. In another possible implementation of the present specification, a data matching method may be used to screen out a target code from multiple candidate codes that have passed the test according to the page type and routing information.

[0069] In an optional embodiment of the present specification, the above-mentioned screening out the target code from the candidate codes according to the page type and the routing information may include the following steps:

[0070] Acquire the candidate code that passes the test, wherein the candidate code that passes the test carries the reference page type and reference routing information;

[0071] Matching the page type with the reference page type, and matching the routing information with the reference routing information to obtain a matching result;

[0072] When the matching result is a successful match, the successfully matched candidate code is determined as the target code.

[0073] It should be noted that if the reference page type and page type of the candidate code that passes the test match, and the reference routing information and routing information match, in the case where both conditions are met, the matching result can be determined as a successful match. If both conditions are not met or only one of them is met, the matching result is determined as a failed match.

[0074] When matching the page type with the reference page type of any candidate code that has passed the test, in one possible implementation, it can be determined whether the reference page type is the same as the page type. If so, it is determined that the two match; if not, it is determined that the two do not match. In another possible implementation, the first similarity between the reference page type and the page type can be calculated. If the first similarity is greater than a first preset threshold, it is determined that the two match. If the first similarity is not greater than the first preset threshold, it is determined that the two do not match, wherein the method for calculating the first similarity includes but is not limited to calculation based on edit distance and cosine similarity technology, and the first preset threshold is specifically set according to actual conditions. The method for matching the routing information with the reference routing information of any candidate code that has passed the test is the same as the method for matching the page type with the reference page type of any candidate code that has passed the test, and the embodiments of this specification will not be repeated.

[0075] In practical applications, there are many ways to obtain candidate codes that pass the test, and the specific selection is based on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, candidate codes that pass the test can be directly obtained from the code library. In another possible implementation of this specification, untested candidate codes generated by the generation model can be obtained, and these candidate codes are tested to obtain candidate codes that pass the test.

[0076] By applying the solution of the embodiments of this specification, a candidate code whose reference routing information and reference page type match the routing information and page type of a target page among multiple candidate codes that have passed the test is determined as the target code, thereby ensuring a high degree of adaptation between the target code and the target page and improving the accuracy of information extraction.

[0077] In an optional embodiment of the present specification, the page information further includes page elements; after the page type is matched with the reference page type and the routing information is matched with the reference routing information, and a matching result is obtained, the following steps may be further included:

[0078] When the matching result is a match failure, the routing information and the page element are input into the first generation model to obtain routing matching information and element location information;

[0079] The code generation prompt information, the route matching information and the element positioning information are input into the second generation model to obtain the target code.

[0080] Specifically, route matching information can be understood as route matching rule information, which is used to identify and locate the target page or the page state of the target page, and can help the server understand the current target page or view state, thereby determining the specific information extraction logic. Element positioning information can be understood as XML path language matching rule information. Element positioning information can use path expressions to locate nodes or node sets in the DOM tree of the target page. The page elements in the target page can be accurately located through element positioning information.

[0081] It should be noted that the candidate codes that pass the test may not include candidate codes whose reference routing information and reference page type match the routing information and page type of the target page. In this case, in order to ensure that the information extraction is carried out normally, the target code can be generated according to the routing information and page elements of the target page, and after the target code is generated, the target code can be tested. If the target code passes the test, the target page is extracted using the target code that passes the test to obtain the information extraction result of the target page. Furthermore, the target code that passes the test can also be stored in the code library for subsequent direct use.

[0082] By applying the solution of the embodiment of this specification, by generating the target code in real time, it is ensured that information can be extracted from the target page, the flexibility of information extraction is improved, and the user experience is enhanced.

[0083] Step 306: Using the target code, extract information from the target page to obtain an information extraction result of the target page.

[0084] In one or more embodiments of the present specification, in response to an information extraction request for a target page, page information of the target page is determined; after the target code is screened out from the candidate codes based on the page type and routing information, the target code can be further used to extract information from the target page to obtain an information extraction result for the target page.

[0085] Specifically, the information extraction result of the target page may be unstructured page content or structured page content, which is specifically determined according to the information extraction request.

[0086] By applying the solution of the embodiment of this specification, the target code is screened out from the candidate codes, and the target code is used to extract information from the target page, thereby realizing automatic information extraction based on the target code and improving the efficiency of information extraction.

[0087] In an optional embodiment of the present specification, the candidate code can be read from other data acquisition devices or databases, or it can be obtained by coding the generation model based on the reference page information of the reference page. Among them, the generation model includes a first generation model and a second generation model. The generation model is used to generate candidate codes that can be used for automatic information extraction based on a given input (reference page information). The generation model can be a large model, or it can be a deep learning model trained based on sample page information of a sample page and sample code corresponding to the sample page information. Before the target code is screened out from the candidate code based on the page type and routing information, the following steps can also be included:

[0088] Acquire a reference page and determine reference page information of the reference page, wherein the reference page information includes reference routing information and reference page elements;

[0089] Inputting the reference route information and the reference page element into the first generation model to obtain reference route matching information and reference element location information;

[0090] The code generation prompt information, the reference route matching information and the reference element positioning information are input into the second generation model to obtain the candidate code corresponding to the reference page.

[0091] Specifically, the reference page refers to a page that carries the reference extraction result. The reference extraction result is the real extraction content included in the reference page, that is, when information is extracted from the reference page, the information extraction result obtained under ideal circumstances should be the same as the reference extraction result. The reference extraction result can be used as a standard for code quality control to ensure that the candidate code generated by the generation model can work properly. The code generation prompt information is a stable in-context learning prompt used to guide the second generation model to accurately generate candidate code.

[0092] Reference route matching information can be understood as reference route matching rule information. Reference route matching information is used to identify and locate reference pages or page states of reference pages, which can help the server understand the current reference page or view state, thereby determining the specific information extraction logic. Reference element positioning information can be understood as reference XML Path Language (Xpath) matching rule information. Reference element positioning information can use path expressions to locate nodes or node sets in the Document Object Model (DOM) tree of the reference page. The page elements in the reference page can be accurately located through the reference element positioning information. In summary, reference route matching information can help the server determine the reference page or part of the reference page to focus on, and reference element positioning information can help the server accurately locate the page elements that need to be captured on the determined reference page.

[0093] In actual applications, there are many ways to obtain reference pages, which can be selected based on actual conditions, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, reference pages can be read from other data acquisition devices or databases. In another possible implementation of this specification, a reference page obtained by manually annotating reference extraction results sent by a client can be received. Furthermore, the implementation method of "determining the reference page information of the reference page" is the same as the implementation method of "determining the page information of the target page" described above, and the embodiments of this specification will not be repeated.

[0094] It is worth noting that when generating candidate codes, you can choose to generate candidate codes in a streaming manner or in a non-streaming manner. Among them, the streaming generation of candidate codes refers to directly generating new candidate codes without caring whether reusable codes have been generated before. Non-streaming generation of candidate codes refers to detecting whether there is reusable code before, and if so, directly reusing the previously generated code as the candidate code. In addition, the candidate code can be HTML code, and during the page processing process, information extraction can be directly based on the HTML of the page.

[0095] By applying the scheme of the embodiment of this specification, the reference routing information and the reference page element are input into the first generation model to obtain the reference routing matching information and the reference element positioning information, thereby ensuring that the generated code can closely fit the structure and logic of the page, and improving the applicability and accuracy of the code; the code generation prompt information, the reference routing matching information and the reference element positioning information are input into the second generation model to obtain the candidate code corresponding to the reference page, so that the code generation is not only based on the static page structure, but also incorporates the dynamic demand context, thereby improving the practicality of the code. By using the generation model to generate the candidate code, the automatic information extraction based on the candidate code is realized, and the efficiency of information extraction is improved.

[0096] In a possible implementation of the present specification, the reference page information further includes model guidance information, and the model guidance information is used to guide the processing of generating the model; the above-mentioned inputting the reference route information and the reference page element into the first generation model to obtain the reference route matching information and the reference element location information may include the following steps:

[0097] Inputting the model guidance information, the reference routing information and the reference page element into the first generation model to obtain the reference routing matching information and the reference element positioning information;

[0098] Inputting the code generation prompt information, the reference route matching information, and the reference element location information into the second generation model to obtain the candidate code corresponding to the reference page may include the following steps:

[0099] The code generation prompt information, the model guidance information, the reference route matching information and the reference element positioning information are input into the second generation model to obtain the candidate code corresponding to the reference page.

[0100] Generally, the model guidance information is obtained by the client marking on the reference page, and the model guidance information includes but is not limited to the type of reference extraction result, the location information of the reference extraction result in the reference page, and the DOM element. By adding the model guidance information to the input of the first generation model and the second generation model respectively, the model processing process can be made more efficient and accurate.

[0101] In an optional embodiment of the present specification, the obtaining of the reference page may include the following steps:

[0102] A reference page sent by a client is received, wherein the reference page is a page obtained by the client by annotating extraction results of a plurality of initial pages of different page types.

[0103] It should be noted that the user can use the client to annotate the extraction results of multiple initial pages of different page types. The annotated page is the reference page, and the annotated extraction result is the reference extraction result carried by the reference page. During the user annotation process, the user can first select the page type, and then slide the left button on the page to mark it, such as marking the title and author on the page.

[0104] By applying the solution of the embodiments of this specification, receiving and utilizing the annotated reference pages, a solid foundation can be laid for subsequent automated code generation, page information extraction, and the like, thereby improving information extraction efficiency.

[0105] In an optional embodiment of the present specification, the above-mentioned determination of the reference page information of the reference page may include the following steps:

[0106] Extracting reference routing information of the reference page from the address bar of the reference page;

[0107] The reference page elements in the reference page are extracted by using a page element extraction strategy, wherein the page element extraction strategy includes at least one of a plug-in extraction strategy, a document object model extraction strategy, and a visual language model extraction strategy.

[0108] It should be noted that the address bar of the reference page refers to the area displayed at the top of the web browser for entering or displaying the full URL of the current web page. The URL contains the information required to access the web page, including the protocol, domain name, route, possible query parameters and fragment identifiers, etc. The plug-in extraction strategy can be called the end-structured strategy. When using the plug-in extraction strategy to extract the reference page elements in the reference page, the front-end plug-in (such as Tumble Monkey) can be used to quickly build a website model to extract the reference page elements in the reference page. The plug-in extraction strategy has the characteristics of fast speed, less manual click operations, and fewer model features. The document object model extraction strategy can be called the DOM reconstruction (Reconstuct) extraction strategy. When using the document object model to extract the reference page elements in the reference page, the website structure can be compressed to obtain the DOM Tree and build a single web page knowledge base for modeling, thereby extracting the reference page elements in the reference page. The document object model extraction strategy can solve the problem that the web page easily exceeds the upper limit of the input of the generation model. The visual language model extraction strategy can be called the VLM structured extraction strategy.

[0109] By applying the solution of the embodiment of this specification, the reference routing information of the reference page is extracted from the address bar of the reference page; the reference page elements in the reference page are extracted using the page element extraction strategy, thereby ensuring the accuracy of the reference page information and further ensuring the accuracy of the candidate code.

[0110] In an optional embodiment of the present specification, the reference page information also includes a reference page type, the reference page carries a reference extraction result, and the reference extraction result is used to test the candidate code; after obtaining the candidate code, the code interpreter can be called based on the calling tool capability of the Agent to test the information extraction capability of the candidate code, that is, after the code generation prompt information, the reference route matching information and the reference element positioning information are input into the second generation model, and the candidate code corresponding to the reference page is obtained, the following steps can also be included:

[0111] Using the candidate codes, information is extracted from the reference page to obtain a predicted extraction result of the reference page;

[0112] Testing the candidate code according to the reference extraction result and the predicted extraction result to obtain a test result;

[0113] When the test result is that the test passes, the candidate code is stored in the code library based on the reference page type and the reference routing information.

[0114] In practical applications, the candidate codes are tested according to the reference extraction results and the predicted extraction results. There are many ways to obtain the test results, which are selected according to the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the reference extraction result and the predicted extraction result can be compared to see if they are the same. If they are the same, the test result is determined to be a passed test; if they are not the same, the test result is determined to be a failed test. In another possible implementation of this specification, the second similarity between the reference extraction result and the predicted extraction result can be calculated. If the second similarity is greater than a second preset threshold, the test is determined to be passed. If the second similarity is not greater than the second preset threshold, the test is determined to be a failed test. The second preset threshold is set specifically according to the actual situation.

[0115] The scheme of the embodiments of this specification is applied to test the candidate code. When the test result is that the test passes, the tested candidate code is stored in the code library based on the reference page type and reference routing information, which helps to quickly find and reuse the candidate code later, especially when facing the same or similar types of pages, page information can be quickly extracted, and by automating the testing of the candidate code, the ability of the candidate code to handle complex pages is improved.

[0116] In an optional embodiment of the present specification, after the candidate code is tested according to the reference extraction result and the predicted extraction result and the test result is obtained, the following steps may be further included:

[0117] If the test result is that the test fails, the test feedback information is sent to the client, and an adjusted reference page sent by the client based on the test feedback information is received;

[0118] Generate candidate codes corresponding to the adjusted reference page, and return to the step of using the candidate codes to extract information from the reference page to obtain the predicted extraction result of the reference page, until the test result of the candidate code is that the test passes, and the candidate code is stored in the code library.

[0119] Specifically, the test feedback information is used to feedback the test failure result to the client and guide the client to send the adjusted reference page. The test feedback information includes but is not limited to error details, specific data points that do not match, and execution logs.

[0120] It should be noted that when the test result of the candidate code is test failure, the reason for the test failure may be that there are certain errors in the reference page and the reference extraction result carried by the reference page. At this time, test feedback information can be sent to the client. The client can correct the element location path, refine the extraction rules or supplement the page context information based on the test feedback information, obtain the adjusted reference page, and send the adjusted reference page to the server.

[0121] Further, after receiving the adjusted reference page sent by the client based on the test feedback information, a candidate code corresponding to the adjusted reference page can be generated. The generation process is the same as the above-mentioned "obtaining a reference page, and determining the reference page information of the reference page, wherein the reference page information includes reference routing information and reference page elements; inputting the reference routing information and the reference page elements into the first generation model to obtain reference routing matching information and reference element positioning information; inputting the code generation prompt information, the reference routing matching information and the reference element positioning information into the second generation model to obtain the candidate code corresponding to the reference page". The reference page actually obtained is the adjusted reference page. The embodiments of this specification will not be described in detail. After generating the candidate code corresponding to the adjusted reference page, the candidate code corresponding to the adjusted reference page can be determined as the candidate code, and the step of using the candidate code to extract information from the reference page and obtain the predicted extraction result of the reference page is returned to execute until the test result of the candidate code is that the test passes, and the candidate code is stored in the code library.

[0122] By applying the solution of the embodiments of this specification, the accuracy and efficiency of automatic code generation are continuously improved through an iterative adjustment mechanism of human-computer interaction, ensuring that the final output code can meet actual application requirements.

[0123] In an optional embodiment of the present specification, after the candidate code is tested according to the reference extraction result and the predicted extraction result and the test result is obtained, the following steps may be further included:

[0124] When the test result is that the test fails, code debugging information is generated;

[0125] Based on the code debugging information, the candidate code is debugged, and the step of using the candidate code to extract information from the reference page and obtain the predicted extraction result of the reference page is returned to be executed, until the test result of the candidate code is that the test passes, and the candidate code is stored in the code library.

[0126] Specifically, code debugging information refers to a series of detailed data and exception causes collected to locate and fix code errors. The causes of code exceptions are mainly exception stack information (Exception Stack Trace), prediction extraction results, and reference extraction results. The exception stack information records in detail the call sequence when the exception occurs, starting from the direct location where the exception occurs, and tracing back to the entry point of the program. It usually includes the exception type, exception message, the code location where the exception occurs, and the call stack.

[0127] In practical applications, there are many ways to debug candidate codes based on code debugging information, which are selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, a code debugging tool is used to debug candidate codes based on code debugging information, wherein the code debugging tool includes but is not limited to a Python debugger (PDb, Python Debugger) and a command line debugger (GDb, GNU Debugger). In another possible implementation of this specification, a generation model can be used to debug candidate codes based on code debugging information. Further, based on the code debugging information, after debugging the candidate code, the debugged candidate code can be determined as the candidate code, and the step of extracting information from the reference page using the candidate code is returned to obtain the predicted extraction result of the reference page until the test result of the candidate code is a test pass, and the candidate code is stored in the code library.

[0128] By applying the solution of the embodiments of this specification, the time required for manual debugging is significantly reduced by automating the debugging process after test failure, so that problems with candidate codes can be quickly located and fixed, the overall efficiency of code development is improved, and it helps to ensure code quality.

[0129] In an optional embodiment of the present specification, the debugging of the candidate code based on the code debugging information may include the following steps:

[0130] Inputting the code debugging information, the reference routing information and the reference page element into the first generation model to obtain the debugged reference routing matching information and the debugged reference element positioning information;

[0131] The code generation prompt information, the reference route matching information after debugging, and the reference element positioning information after debugging are input into the second generation model to obtain the candidate code after debugging.

[0132] It should be noted that the implementation method of "inputting code debugging information, reference routing information and reference page elements into the first generation model to obtain reference routing matching information after debugging and reference element positioning information after debugging; inputting code generation prompt information, reference routing matching information after debugging and reference element positioning information after debugging into the second generation model to obtain candidate code after debugging" is the same as the implementation method of "inputting reference routing information and reference page elements into the first generation model to obtain reference routing matching information and reference element positioning information; inputting code generation prompt information, reference routing matching information and reference element positioning information into the second generation model to obtain candidate code corresponding to the reference page", and will not be repeated in the embodiments of this specification.

[0133] By applying the solution of the embodiments of this specification, the code debugging information is fed back to the generation model, and the generation model automatically iterates and modifies the candidate code, the speed of problem identification and repair can be significantly accelerated, and the time spent by developers on repetitive work can be reduced.

[0134] The following combination Figure 4 , taking the application of the information extraction method provided in this specification in the page structure extraction scenario as an example, the information extraction method is further described. Figure 4 A flow chart of a page structure extraction method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0135] Step 402: In response to a page structure extraction request for a target page, determine page information of the target page, wherein the page information includes a page type and routing information.

[0136] Step 404: Filter out target codes from candidate codes according to the page type and routing information, wherein the candidate codes are compiled based on reference page information of the reference page.

[0137] Step 406: Using the target code, extract structured information from the target page to obtain a page structure extraction result of the target page.

[0138] It should be noted that the implementation of step 402 to step 406 is the same as the implementation of step 302 to step 306 described above, and will not be described in detail in the embodiments of this specification.

[0139] By applying the solution of the embodiment of this specification, the target code is screened out from the candidate codes, and the target code is used to extract structured information of the target page, thereby realizing automatic structured information extraction based on the target code and improving the efficiency of structured information extraction.

[0140] See also Figure 5 , Figure 5 A flowchart of a processing process of an information extraction method provided by an embodiment of the present specification is shown. The processing process of the information extraction method specifically includes two stages: candidate code generation and page information extraction. Next, these two stages are described in detail.

[0141] Candidate code generation: obtain the reference page by annotating the reference extraction result performed by the user; determine the reference routing information of the reference page; use the page element extraction strategy to extract the reference page elements in the reference page, wherein the page element extraction strategy includes at least one of the plug-in extraction strategy, the document object model extraction strategy, and the visual language model extraction strategy; input the model guidance information, the reference routing information, and the reference page elements into the first generation model to obtain the reference routing matching information and the reference element positioning information; input the code generation prompt information, the reference routing matching information, and the reference element positioning information into the second generation model to obtain the candidate code corresponding to the reference page; test the candidate code in the docker container to obtain the predicted extraction result of the reference page; and Test the extraction result and determine the test result of the candidate code; determine whether the test result passes: if so, based on the reference page type and reference routing information, store the candidate code in the code library and enter the page information extraction stage; if not, debug the candidate code in at least one of the following ways: first, send the test feedback information to the client, and the user re-marks the reference extraction result based on the test feedback information to obtain an adjusted reference page; receive the adjusted reference page sent by the client; generate the candidate code corresponding to the adjusted reference page, and return to execute the step of testing the candidate code in the docker container to obtain the predicted extraction result of the reference page, until the test result of the candidate code is that the test passes, store the candidate code in the code library and enter the page information extraction stage. The second method is to generate code debugging information; input the code debugging information, reference route information and reference page elements into the first generation model to obtain reference route matching information after debugging and reference element positioning information after debugging; input the code generation prompt information, reference route matching information after debugging and reference element positioning information after debugging into the second generation model to obtain candidate code after debugging; return to execute the step of testing the candidate code in the docker container to obtain the predicted extraction result of the reference page, until the test result of the candidate code is that the test passes, store the candidate code in the code library, and enter the page information extraction stage;

[0142] Page information extraction: In response to an information extraction request for a target page, determine the page information of the target page; select the target code from multiple candidate codes that have passed the test based on the page type and routing information; use the target code to extract information from the target page to obtain the information extraction result of the target page.

[0143] Corresponding to the above-mentioned information extraction method embodiment, this specification also provides an information extraction device embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of an information extraction device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0144] A first determination module 602 is configured to determine page information of a target page in response to an information extraction request for the target page, wherein the page information includes a page type and routing information;

[0145] A first screening module 604 is configured to screen target codes from candidate codes according to page type and routing information, wherein the candidate codes are obtained by coding based on reference page information of the reference page;

[0146] The first extraction module 606 is configured to extract information from the target page using the target code to obtain an information extraction result of the target page.

[0147] Optionally, the device also includes: an acquisition module, configured to acquire a reference page and determine reference page information of the reference page, wherein the reference page information includes reference routing information and reference page elements; input the reference routing information and reference page elements into a first generation model to obtain reference routing matching information and reference element positioning information; input the code generation prompt information, reference routing matching information and reference element positioning information into a second generation model to obtain candidate codes corresponding to the reference page.

[0148] Optionally, the reference page information also includes a reference page type, the reference page carries a reference extraction result, and the reference extraction result is used to test the candidate code; the device also includes: a testing module, configured to use the candidate code to extract information from the reference page to obtain a predicted extraction result of the reference page; test the candidate code according to the reference extraction result and the predicted extraction result to obtain a test result; when the test result is that the test passes, the candidate code is stored in a code library based on the reference page type and the reference routing information.

[0149] Optionally, the device also includes: a generation module, configured to send test feedback information to the client when the test result is that the test fails, and receive an adjusted reference page sent by the client based on the test feedback information; generate a candidate code corresponding to the adjusted reference page, and return to execute the step of using the candidate code to extract information from the reference page to obtain a predicted extraction result of the reference page, until the test result of the candidate code is that the test passes, and the candidate code is stored in the code library.

[0150] Optionally, the device also includes: a debugging module, configured to generate code debugging information when the test result is that the test fails; based on the code debugging information, debug the candidate code, and return to execute the step of using the candidate code to extract information from the reference page to obtain the predicted extraction result of the reference page, until the test result of the candidate code is that the test passes, and the candidate code is stored in the code library.

[0151] Optionally, the debugging module is further configured to input code debugging information, reference route information and reference page elements into the first generation model to obtain reference route matching information after debugging and reference element positioning information after debugging; and input code generation prompt information, reference route matching information after debugging and reference element positioning information after debugging into the second generation model to obtain candidate code after debugging.

[0152] Optionally, the acquisition module is further configured to extract reference routing information of the reference page from the address bar of the reference page; and extract reference page elements in the reference page using a page element extraction strategy, wherein the page element extraction strategy includes at least one of a plug-in extraction strategy, a document object model extraction strategy, and a visual language model extraction strategy.

[0153] Optionally, the acquisition module is further configured to receive a reference page sent by a client, wherein the reference page is a page obtained by the client by annotating extraction results of a plurality of initial pages of different page types.

[0154] Optionally, the first screening module 604 is further configured to obtain candidate codes that pass the test, wherein the candidate codes that pass the test carry reference page types and reference routing information; match the page type with the reference page type, and match the routing information with the reference routing information to obtain a matching result; and when the matching result is a successful match, determine the successfully matched candidate code as the target code.

[0155] Optionally, the page information also includes page elements; the device also includes: an input module, which is configured to input the routing information and page elements into the first generation model to obtain routing matching information and element positioning information when the matching result is a matching failure; and input the code generation prompt information, routing matching information and element positioning information into the second generation model to obtain the target code.

[0156] By applying the solution of the embodiment of this specification, the target code is screened out from the candidate codes, and the target code is used to extract information from the target page, thereby realizing automatic information extraction based on the target code and improving the efficiency of information extraction.

[0157] The above is a schematic scheme of an information extraction device of this embodiment. It should be noted that the technical scheme of the information extraction device and the technical scheme of the above information extraction method belong to the same concept, and the details not described in detail in the technical scheme of the information extraction device can be referred to the description of the technical scheme of the above information extraction method.

[0158] Corresponding to the above-mentioned page structure extraction method embodiment, this specification also provides a page structure extraction device embodiment, Figure 7 FIG. 1 shows a schematic diagram of a structured page extraction device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0159] The second determination module 702 is configured to determine page information of the target page in response to the page structure extraction request for the target page, wherein the page information includes page type and routing information;

[0160] The second screening module 704 is configured to screen the target code from the candidate codes according to the page type and the routing information, wherein the candidate code is obtained by coding based on the reference page information of the reference page;

[0161] The second extraction module 706 is configured to extract structured information from the target page using the target code to obtain a page structure extraction result of the target page.

[0162] By applying the solution of the embodiment of this specification, the target code is screened out from the candidate codes, and the target code is used to extract structured information of the target page, thereby realizing automatic structured information extraction based on the target code and improving the efficiency of structured information extraction.

[0163] The above is a schematic scheme of a page structure extraction device of this embodiment. It should be noted that the technical scheme of the page structure extraction device and the technical scheme of the above-mentioned page structure extraction method belong to the same concept, and the details not described in detail in the technical scheme of the page structure extraction device can be found in the description of the technical scheme of the above-mentioned page structure extraction method.

[0164] Figure 8 The structure block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and the database 850 is used to store data.

[0165] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0166] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0167] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.

[0168] The processor 820 is used to execute a computer program / instruction, which implements the steps of the above-mentioned information extraction method or page structure extraction method when executed by the processor.

[0169] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned information extraction method and page structure extraction method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned information extraction method or page structure extraction method.

[0170] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned information extraction method or page structure extraction method when executed by a processor.

[0171] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned information extraction method and page structure extraction method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned information extraction method or page structure extraction method.

[0172] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned information extraction method or page structure extraction method when executed by a processor.

[0173] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned information extraction method and page structure extraction method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned information extraction method or page structure extraction method.

[0174] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0176] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0177] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0178] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An information extraction method, comprising: In response to an information extraction request for a target page, determining page information of the target page, wherein the page information includes a page type and routing information; Filtering target codes from candidate codes according to the page type and the routing information, wherein the candidate codes are obtained by coding based on reference page information of the reference page; The target code is used to extract information from the target page to obtain an information extraction result of the target page.

2. The method according to claim 1, before filtering out the target code from the candidate codes according to the page type and the routing information, further comprises: Acquire a reference page, and determine reference page information of the reference page, wherein the reference page information includes reference routing information and reference page elements; Inputting the reference route information and the reference page element into a first generation model to obtain reference route matching information and reference element location information; The code generation prompt information, the reference route matching information and the reference element location information are input into a second generation model to obtain a candidate code corresponding to the reference page.

3. The method according to claim 2, wherein the reference page information further includes a reference page type, the reference page carries a reference extraction result, and the reference extraction result is used to test the candidate code; After inputting the code generation prompt information, the reference route matching information and the reference element location information into the second generation model to obtain the candidate code corresponding to the reference page, the method further includes: Using the candidate code, extract information from the reference page to obtain a predicted extraction result of the reference page; Testing the candidate code according to the reference extraction result and the predicted extraction result to obtain a test result; When the test result is that the test passes, the candidate code is stored in a code library based on the reference page type and the reference routing information.

4. The method according to claim 3, wherein the candidate code is tested according to the reference extraction result and the predicted extraction result, and after obtaining the test result, the method further comprises: If the test result is that the test fails, sending test feedback information to the client, and receiving an adjusted reference page sent by the client based on the test feedback information; Generate a candidate code corresponding to the adjusted reference page, and return to execute the step of using the candidate code to extract information from the reference page to obtain a predicted extraction result of the reference page, until the test result of the candidate code is that the test passes, and store the candidate code in the code library.

5. The method according to claim 3, wherein the candidate code is tested according to the reference extraction result and the predicted extraction result, and after obtaining the test result, the method further comprises: If the test result is that the test fails, generating code debugging information; Based on the code debugging information, the candidate code is debugged, and the step of using the candidate code to extract information from the reference page to obtain a predicted extraction result of the reference page is returned to execute, until the test result of the candidate code is that the test passes, and the candidate code is stored in the code library.

6. The method according to claim 5, wherein the debugging of the candidate code based on the code debugging information comprises: Inputting the code debugging information, the reference routing information and the reference page element into the first generation model to obtain debugged reference routing matching information and debugged reference element location information; The code generation prompt information, the reference route matching information after debugging and the reference element location information after debugging are input into the second generation model to obtain the candidate code after debugging.

7. The method according to claim 2, wherein determining the reference page information of the reference page comprises: Extracting reference routing information of the reference page from the address bar of the reference page; The reference page elements in the reference page are extracted using a page element extraction strategy, wherein the page element extraction strategy includes at least one of a plug-in extraction strategy, a document object model extraction strategy, and a visual language model extraction strategy.

8. The method according to claim 2, wherein obtaining a reference page comprises: A reference page sent by a client is received, wherein the reference page is a page obtained by the client by annotating extraction results of a plurality of initial pages of different page types.

9. The method according to claim 1, wherein the step of selecting a target code from candidate codes according to the page type and the routing information comprises: Acquire a candidate code that passes the test, wherein the candidate code that passes the test carries a reference page type and reference routing information; Matching the page type with the reference page type, and matching the routing information with the reference routing information to obtain a matching result; When the matching result is a successful match, the successfully matched candidate code is determined as the target code.

10. The method according to claim 9, wherein the page information further comprises page elements; After matching the page type with the reference page type and matching the routing information with the reference routing information to obtain a matching result, the method further includes: When the matching result is a match failure, inputting the routing information and the page element into a first generation model to obtain routing matching information and element location information; The code generation prompt information, the route matching information and the element location information are input into a second generation model to obtain the target code.

11. A page structure extraction method, comprising: In response to a page structure extraction request for a target page, determining page information of the target page, wherein the page information includes a page type and routing information; Filtering target codes from candidate codes according to the page type and the routing information, wherein the candidate codes are obtained by coding based on reference page information of the reference page; The target code is used to extract structured information of the target page to obtain a page structure extraction result of the target page.

12. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 10 or claim 11 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method described in any one of claims 1 to 10 or claim 11.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10 or claim 11.

Citation Information

Patent Citations

  • Automatic extraction method of web page information based on WEB structure

    CN101727498A

  • Webpage information extraction method and device thereof

    CN101944094A

  • Method and equipment used for determining page type information of target pages

    CN103577447A