Information extraction method and system using large model

Abstract

The invention provides an information extraction method and system using a large model. The method comprises the following steps: extracting a plurality of schemes of attributes by using the large model; the large model converts each scheme into a corresponding code through a natural language algorithm; analyzing the webpage by using the large model, and extracting truth values of attributes; applying the code corresponding to each scheme to the webpage to obtain a value to be verified; comparing the value to be verified with the true value of the attribute, and filtering out incorrect schemes; according to the method, multiple schemes for extracting the attributes can be generated through the large model, corresponding codes are generated for each scheme for implementation, then the truth values of the attributes are extracted from the webpage through the large model to serve as references, incorrect schemes are filtered out, the efficiency, accuracy and robustness of information extraction can be improved, and manpower resources and time cost are saved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an information extraction method and system using a large model. Background Art

[0002] Public web pages contain a large amount of public information, such as product prices, addresses, and phone numbers. This information is very valuable for areas such as information retrieval and recommendation.

[0003] At present, there are two traditional methods for information extraction: 1) Information extraction through manual writing of rules and templates, which requires a lot of manpower and material resources, and it is difficult to adapt to different websites and pages. 2) Information extraction based on machine learning methods, which requires a large amount of labeled data and manpower for model training. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an information extraction method and system using a large model, which can use the large model to generate multiple schemes for extracting attributes, and generate corresponding code implementations for each scheme, and then use the large model to extract the true value of the attribute from the web page as a reference to filter out incorrect schemes, thereby improving the efficiency, accuracy and robustness of information extraction and saving human resources and time costs.

[0005] In a first aspect, an embodiment of the present invention provides an information extraction method using a large model, the method comprising:

[0006] A plurality of schemes for extracting attributes using the large model;

[0007] The large model converts each of the solutions into corresponding codes through a natural language algorithm;

[0008] Parsing the web page using the large model to extract the true value of the attribute;

[0009] Apply the code corresponding to each of the schemes to the webpage to obtain a value to be verified;

[0010] The value to be verified is compared with the true value of the attribute to filter out incorrect solutions.

[0011] Furthermore, the large model is an NLP model, which is a neural network model trained using a deep learning algorithm; the multiple schemes of the attributes include regular expressions, CSS selectors, and Xpath expressions.

[0012] Furthermore, multiple schemes for extracting attributes using the large model include:

[0013] The large model is used to learn the features of different websites and pages, extract the attributes, and generate multiple solutions for the attributes.

[0014] Furthermore, the method further comprises:

[0015] A voting scheme is used to select the value with the largest number of identical values ​​from the values ​​to be verified;

[0016] The value with the largest number of identical values ​​is taken as the true value.

[0017] Furthermore, the value to be verified is compared with the true value of the attribute to filter out incorrect solutions, including:

[0018] The value to be verified is compared with the true value of the attribute, and incorrect regular expressions, incorrect CSS selectors or incorrect XPath expressions are filtered out.

[0019] In a second aspect, an embodiment of the present invention provides an information extraction system using a large model, the system comprising:

[0020] An extraction module, for extracting multiple schemes of attributes using the large model;

[0021] A conversion module, used for the large model to convert each of the solutions into corresponding codes through a natural language algorithm;

[0022] A parsing module, used to parse the web page using the large model and extract the true value of the attribute;

[0023] An application module, used for applying the code corresponding to each of the schemes to the webpage to obtain a value to be verified;

[0024] The comparison module is used to compare the value to be verified with the true value of the attribute and filter out incorrect solutions.

[0025] Furthermore, the large model is an NLP model, which is a neural network model trained using a deep learning algorithm; the multiple schemes of the attributes include regular expressions, CSS selectors, and Xpath expressions.

[0026] Furthermore, the extraction module is specifically used for:

[0027] The large model is used to learn the features of different websites and pages, extract the attributes, and generate multiple solutions for the attributes.

[0028] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the method described above when executing the computer program.

[0029] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the method as described above.

[0030] The embodiment of the present invention provides an information extraction method and system using a big model, including: using the big model to extract multiple schemes of attributes; the big model converts each scheme into a corresponding code through a natural language algorithm; using the big model to parse a web page to extract the true value of the attribute; applying the code corresponding to each scheme to the web page to obtain a value to be verified; comparing the value to be verified with the true value of the attribute to filter out incorrect schemes; the big model can be used to generate multiple schemes for extracting attributes, and corresponding code implementations can be generated for each scheme, and then the big model is used to extract the true value of the attribute from the web page as a reference to filter out incorrect schemes, which can improve the efficiency, accuracy and robustness of information extraction and save human resources and time costs.

[0031] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A flow chart of the information extraction method using a large model provided in the first embodiment of the present invention;

[0035] Figure 2 A schematic diagram of an information extraction system using a large model provided in Example 2 of the present invention.

[0036] icon:

[0037] 1-Extraction module; 2-Transformation module; 3-Analysis module; 4-Application module; 5-Comparison module. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] To facilitate understanding of this embodiment, the embodiment of the present invention is described in detail below.

[0040] Embodiment 1:

[0041] Figure 1 This is a flow chart of the information extraction method using a large model provided in Example 1 of the present invention.

[0042] Reference Figure 1 , the method comprises the following steps:

[0043] Step S101, extracting multiple schemes of attributes using a large model;

[0044] Here, using multiple schemes of extracting attributes from a large model can overcome the shortcomings of traditional methods that require manual writing of rules and templates, and also avoid the problem that machine learning methods require a large amount of labeled data and training time.

[0045] For an article describing an athlete, it is necessary to extract attributes such as "height" and "date of birth" due to business needs. Generally, the specific attributes are determined by business needs. Here, ChatGPT is used as a large model for explanation. Business requirements: Generate a regular expression and extract all the following phone numbers; Note: Only the regular expression is returned. If it cannot be implemented, None is returned.

[0046] The specific content is: mobile phone: 1580077, from website A; mobile phone: 1391774, from website B; mobile phone: 1381793, from website C; mobile phone 1365161, from website D. Therefore, the regular expression that can match all the phone numbers in the above text is obtained: 1[3-9]\d{9}.

[0047] Step S102, the big model converts each solution into corresponding code through a natural language algorithm;

[0048] Here, the code is generated through a natural language algorithm, that is, each solution is used as the input of the natural language algorithm, and the code corresponding to each solution is output. This can automatically generate the code and avoid the errors and tediousness of manual code writing.

[0049] Step S103, using the large model to parse the web page and extract the true value of the attribute;

[0050] Here, the big model is used to extract the true value of the attribute to be extracted from the web page as a reference. The purpose of this step is to obtain the true value of the attribute in the web page for comparison, which can avoid the problem of manually annotating data and accurately obtain the true value of the attribute.

[0051] Specifically, ChatGPT is used as a large model for explanation. Business requirement: extract all phone numbers that appear below. Note: only return results. If this cannot be achieved, None is returned.

[0052] The specific content is: The following are all the telephone numbers that appear in the text: 1580077, 1391774, 1381793, 1365161.

[0053] Step S104, applying the code corresponding to each solution to the web page to obtain the value to be verified;

[0054] Here, the code corresponding to each solution is applied to the web page to obtain the value to be verified, which can be compared and screened to obtain the correct implementation solution.

[0055] Step S105, comparing the value to be verified with the true value of the attribute, and filtering out incorrect solutions.

[0056] Specifically, through comparison, the correct implementation scheme can be screened out, incorrect schemes can be filtered out, and the accuracy of the extracted attribute values ​​can be ensured.

[0057] Furthermore, the large model is an NLP model, which is a neural network model trained using a deep learning algorithm; the multiple schemes of attributes include regular expressions, CSS selectors, and Xpath expressions.

[0058] Here, NLP models usually use the Transformer architecture, such as BERT and GPT-3, which can perform high-level understanding and processing of natural language, such as text classification, named entity recognition, machine translation, and question-answering systems.

[0059] Furthermore, step S101 includes:

[0060] Use the big model to learn the features of different websites and pages, extract attributes, and generate multiple schemes of attributes.

[0061] Furthermore, the method further comprises the following steps:

[0062] Step S201, using a voting scheme to select the value with the largest number of identical values ​​from the values ​​to be verified;

[0063] Step S202, taking the value with the largest number of identical values ​​as the true value.

[0064] Here, the purpose of this step is to further improve the accuracy of the extracted attribute values. By adopting a voting scheme, the error of a single scheme can be avoided, thereby obtaining a more accurate extraction result.

[0065] Specifically, the value to be verified is compared with the true value of the attribute to filter out incorrect solutions. Here we take a regular expression as an example:

[0066] 1) Use regular expression solution A1 to calculate data B1. If the "true value" results of A1 and B1 are inconsistent, the number of failures of A1 increases by one;

[0067] 2) Similarly, match A1 with all data B to obtain the total number of failures of A1;

[0068] 3) Similarly, calculate all regular expression solutions A and the total number of failures of each A solution;

[0069] 4) Rank the solutions from high to low by the number of failures, and check the top 50% with the highest number of failures. For example, there are 100 A solutions in total, 70 of which have failed, and the remaining 30 have no failures. The 35 solutions with the highest number of failures of these 70 solutions are filtered out (rounded down), and finally 35+30=65 A solutions are left.

[0070] In addition, in order to obtain more accurate extraction results, a voting scheme is used to select the value with the most identical values ​​from the values ​​to be verified; the value with the most identical values ​​is taken as the true value. For example, there is an unknown data B', B' does not use a large model to directly take the true value (cost considerations), and 65 A schemes are used at this time.

[0071] 1) All 65 A solutions are applied to B', and 65 values ​​to be verified are obtained;

[0072] 2) Take the mode of the 65 values ​​to be verified as the true value;

[0073] 3) If the 65 values ​​to be verified are all different, use the large model to directly get the true value.

[0074] Further, step S105 includes:

[0075] Compare the value to be validated with the true value of the attribute to filter out incorrect regular expressions, incorrect CSS selectors, or incorrect XPath expressions.

[0076] This application uses a large model to efficiently generate extraction solutions and code implementations, which can greatly save time and resource consumption compared to manually writing code. The process of using the generated code to extract information is also very efficient, and can quickly process a large number of public web pages, improving the efficiency of information extraction.

[0077] The large model has strong expressive and generalization capabilities and can adapt to various types of public web pages and different information extraction tasks. Since the process of generating extraction schemes and code implementation is automated, it can avoid errors and deviations caused by human factors and improve the accuracy and robustness of the extraction results.

[0078] In the process of extracting information using a large model, only a small amount of manpower is required to complete the entire process. Compared with manually writing code and manually verifying the extraction results, it greatly saves human resources and time costs, making the cost of information extraction lower.

[0079] The extraction scheme and code implementation generated by the large model can quickly, efficiently and accurately extract the required information from public web pages, with high extraction accuracy and effect guarantee. At the same time, incorrect implementation schemes are filtered out by voting to ensure the correctness and reliability of the extraction results.

[0080] The embodiment of the present invention provides an information extraction method using a big model, including: using the big model to extract multiple schemes of attributes; the big model converts each scheme into a corresponding code through a natural language algorithm; using the big model to parse a web page to extract the true value of the attribute; applying the code corresponding to each scheme to the web page to obtain a value to be verified; comparing the value to be verified with the true value of the attribute to filter out incorrect schemes; the big model can be used to generate multiple schemes for extracting attributes, and corresponding code implementations can be generated for each scheme, and then the big model is used to extract the true value of the attribute from the web page as a reference to filter out incorrect schemes, which can improve the efficiency, accuracy and robustness of extracting information and save human resources and time costs.

[0081] Embodiment 2:

[0082] Figure 2 Schematic diagram of an information extraction system using a large model provided in Embodiment 2 of the present invention.

[0083] Reference Figure 2 , the system comprises:

[0084] Extraction module 1, used to extract multiple schemes of attributes using a large model;

[0085] Conversion module 2, used by the large model to convert each solution into corresponding code through natural language algorithm;

[0086] Parsing module 3, used to parse the web page using the large model and extract the true value of the attribute;

[0087] Application module 4, used for applying the code corresponding to each solution to the web page to obtain the value to be verified;

[0088] The comparison module 5 is used to compare the value to be verified with the true value of the attribute and filter out incorrect solutions.

[0089] Furthermore, the large model is an NLP model, which is a neural network model trained using a deep learning algorithm; the multiple schemes of attributes include regular expressions, CSS selectors, and Xpath expressions.

[0090] Furthermore, the extraction module 1 is specifically used for:

[0091] Use the big model to learn the features of different websites and pages, extract attributes, and generate multiple schemes of attributes.

[0092] Furthermore, the system also includes:

[0093] A selection module (not shown), configured to select the value with the largest number of identical values ​​from the values ​​to be verified by using a voting scheme;

[0094] As a module (not shown), it is used to take the value with the largest number of identical values ​​as the true value.

[0095] Furthermore, the comparison module 5 is specifically used for:

[0096] Compare the value to be validated with the true value of the attribute to filter out incorrect regular expressions, incorrect CSS selectors, or incorrect XPath expressions.

[0097] The embodiment of the present invention provides an information extraction system using a large model, including: using the large model to extract multiple schemes of attributes; the large model converts each scheme into a corresponding code through a natural language algorithm; using the large model to parse a web page to extract the true value of the attribute; applying the code corresponding to each scheme to the web page to obtain a value to be verified; comparing the value to be verified with the true value of the attribute to filter out incorrect schemes; the large model can be used to generate multiple schemes for extracting attributes, and corresponding code implementations can be generated for each scheme, and then the large model is used to extract the true value of the attribute from the web page as a reference to filter out incorrect schemes, which can improve the efficiency, accuracy and robustness of extracting information and save human resources and time costs.

[0098] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the information extraction method using a large model provided in the above embodiment are implemented.

[0099] An embodiment of the present invention further provides a computer-readable medium having a non-volatile program code executable by a processor, wherein a computer program is stored on the computer-readable medium, and when the computer program is executed by the processor, the steps of the method of the above embodiment are executed.

[0100] The computer program product provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0102] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0103] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0104] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0105] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for information extraction using large models, It is characterized in that The method comprises: A plurality of schemes for extracting attributes using the large model; The large model converts each of the solutions into corresponding codes through a natural language algorithm; Parsing the web page using the large model to extract the true value of the attribute; Apply the code corresponding to each of the schemes to the webpage to obtain a value to be verified; The value to be verified is compared with the true value of the attribute to filter out incorrect solutions.

2. The information extraction method using a large model according to claim 1, It is characterized in that The large model is an NLP model, which is a neural network model trained using a deep learning algorithm; the multiple schemes of the attributes include regular expressions, CSS selectors, and Xpath expressions.

3. The information extraction method using a large model according to claim 2, It is characterized in that There are multiple schemes for extracting attributes using the large model, including: The large model is used to learn the features of different websites and pages, extract the attributes, and generate multiple solutions for the attributes.

4. The information extraction method using a large model according to claim 1, It is characterized in that The method further comprises: A voting scheme is used to select the value with the largest number of identical values ​​from the values ​​to be verified; The value with the largest number of identical values ​​is taken as the true value.

5. The information extraction method using a large model according to claim 1, It is characterized in that Compare the value to be verified with the true value of the attribute and filter out incorrect solutions, including: The value to be verified is compared with the true value of the attribute, and incorrect regular expressions, incorrect CSS selectors or incorrect XPath expressions are filtered out.

6. An information extraction system using a large model, It is characterized in that The system comprises: An extraction module, for extracting multiple schemes of attributes using the large model; A conversion module, used for the large model to convert each of the solutions into corresponding codes through a natural language algorithm; A parsing module, used to parse the web page using the large model and extract the true value of the attribute; An application module, used for applying the code corresponding to each of the schemes to the webpage to obtain a value to be verified; The comparison module is used to compare the value to be verified with the true value of the attribute and filter out incorrect solutions.

7. The information extraction system using a large model according to claim 6, It is characterized in that The large model is an NLP model, which is a neural network model trained using a deep learning algorithm; the multiple schemes of the attributes include regular expressions, CSS selectors, and Xpath expressions.

8. The information extraction system using a large model according to claim 7, It is characterized in that The extraction module is specifically used for: The large model is used to learn the features of different websites and pages, extract the attributes, and generate multiple solutions for the attributes.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor. It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, It is characterized in that The program code enables the processor to execute the method according to any one of claims 1 to 5.

Images