AI corpus data acquisition method and equipment based on engine search and storage medium

Abstract

The invention discloses an AI corpus data acquisition method and device based on engine search and a storage medium, and relates to the technical field of corpus data acquisition, and the method comprises the following steps: determining keywords and terms related to a target corpus, and expanding a keyword list by using synonyms and related terms; searching based on the keyword list to obtain searched webpages, and extracting effective webpages in the searched webpages based on the duplicate checking rate; performing data cleaning on the captured webpage content to remove irrelevant information and repeated information; setting a duplicate checking test experiment, analyzing a sample duplicate checking rate through the sample webpage, and combining data cleaning analysis to obtain a duplicate checking threshold value; the method and the device are used for solving the problems that an existing corpus data acquisition technology is not accurate enough in extraction of a search webpage, so that after a long-time complex processing process, the extracted corpus data is less, and resources are wasted.

Description

Technical Field

[0001] The present invention relates to the technical field of corpus data acquisition, and specifically to an AI corpus data acquisition method, device and storage medium based on engine search. Background Art

[0002] Corpus data acquisition technology refers to the process of using various methods and tools to collect and obtain text data required for linguistic research from different sources. These technologies can cover methods of collecting data from various sources such as the Internet, digital libraries, social media, books, and news releases. The development of corpus data acquisition technology enables researchers to obtain large-scale language data more easily, thereby conducting research in fields such as linguistic analysis, machine learning, and natural language processing.

[0003] The existing corpus data acquisition technology usually crawls web pages through search, removes advertising web pages, extracts content from all web pages and removes duplicates to obtain corpus data. This method has defects. Since keywords are used for search, the content difference of the searched web pages will not be too large. At this time, all of them are extracted, and web pages with high repetition will be extracted. After deduplication, the actual corpus data extracted is very small or even no valid corpus data. On the one hand, such web pages increase the duration and complexity of the processing process, and on the other hand, they contribute less to the acquisition of corpus data. For example, in the patent application with publication number CN112818212A, a corpus data collection method, device, computer equipment and storage medium are disclosed. The solution is to perform a comprehensive analysis on the searched web pages, which include many repeated web pages and web pages with high repetition. Less valid corpus data is extracted, and the complexity and processing time of the processing are increased. The existing corpus data acquisition technology also has the problem of not being accurate enough in extracting the searched web pages, resulting in less corpus data extracted after a long and complex processing process, resulting in a waste of resources. Summary of the invention

[0004] The present invention aims to provide an AI corpus data acquisition method based on engine search to solve the problem that the existing corpus data acquisition technology in the prior art is not accurate enough in extracting search web pages, resulting in less corpus data extracted after a long and complex processing process, resulting in a waste of resources.

[0005] To achieve the above objectives, in a first aspect, the present application provides an AI corpus data acquisition method based on engine search, comprising the following steps: Identify keywords and terms relevant to the target corpus and expand the keyword list using synonyms and related terms; Based on the keyword list, a search webpage is obtained, the webpage content of the search webpage is crawled, the duplicate rate between the webpage contents is analyzed, and valid webpages in the search webpage are extracted based on the duplicate rate; Clean the captured webpage content to remove irrelevant and duplicate information; Set up a duplicate checking test experiment, analyze the sample duplicate checking rate through sample web pages, and combine data cleaning analysis to obtain the duplicate checking threshold to provide a judgment standard for valid web pages.

[0006] Further, determining keywords and terms related to the target corpus and expanding the keyword list using synonyms and related terms includes the following sub-steps: The target corpus is the corpus data that needs to be acquired; When you need to obtain corpus data in any field, mark the keywords related to this field as target keywords, and mark the terms related to this field as related terms; For the target keyword, search for synonyms of the target keyword through the thesaurus and mark them as synonymous keywords; The keyword list consists of the target keyword, synonymous keywords, and related terms.

[0007] Further, obtaining a search webpage based on a keyword list search, crawling the webpage content of the search webpage, analyzing the duplication rate between the webpage contents, and extracting valid webpages in the search webpage based on the duplication rate includes the following sub-steps: The target keywords, synonymous keywords and related terms in the keyword list are collectively referred to as search elements, and the search elements are searched through different search engine APIs; Advertisement pages in search results are blocked by ad blocking scripts, only non-advertisement pages in search results are obtained, all pages in search results are obtained, and marked as search pages; After obtaining the first number of search pages, the search pages are numbered and represented by symbols W. n Represents, where n∈Z+, and n is the serial number of W, and Z+ represents a positive integer; Extract W n The text content in the text box is marked as T. n , set subscripts i and j, where both i and j are positive integers and the value range of i and j is the same as that of n, but j≠i; Starting from i=1, T i With every T j Perform duplicate analysis and obtain the duplicate rate, which is represented by the symbol C(i,j), where C(i,j) represents T j With T i The duplication rate of Analyze C(i,j) and filter out valid web pages in the search web pages.

[0008] Further, analyzing C(i,j) to screen valid web pages in the search web pages includes the following sub-steps: Compare C(i,j) with the duplicate checking threshold. If C(i,j) is less than the duplicate checking threshold, output a pending signal for the webpage; if C(i,j) is greater than or equal to the duplicate checking threshold, output an invalid signal for the webpage. If the output web page is pending, temporarily retain T j If the webpage is invalid, j in C(i,j) is included in the invalid range, and the invalid range is represented by h; i+1 is added to T i With every T j Perform duplicate analysis. At this time, j≠i≠h. If i+1 is equal to any value in h, then i is increased by 1 again. Continue analyzing until i equals the maximum value of n, and then mark the values ​​in the non-intersection of n and h as g, and search for web page W. g Mark as a valid page.

[0009] Furthermore, data cleaning is performed on the captured webpage content to remove irrelevant and duplicate information, including the following sub-steps: In the front-end design of web pages, web pages are usually divided into the title part, the navigation part and the text part. The text part of the effective web page is extracted; Mark the text content between the first line indentation format and the line break in the body as the body content, and number the body content, represented by the symbol F(g,d), where d∈Z+, and (g,d) is the serial number of F, and F(g,d) represents the valid web page W g The content of Article d in the text; Count all F(g,d), and compare the content of each F(g,d). If there are F(g,d) with exactly the same content, remove the content of one of them, and the removed content will no longer participate in the content comparison. After the content comparison and analysis is completed, the remaining text content is marked as target corpus and stored in the AI ​​corpus.

[0010] Furthermore, a duplicate checking test experiment is set up to analyze the sample duplicate checking rate through sample web pages, and the duplicate checking threshold is obtained by combining data cleaning analysis to provide a judgment standard for valid web pages, including the following sub-steps: Obtain sample web pages, analyze the sample duplication rate of the sample web pages, and extract the number of target corpora in the sample web pages through data cleaning to obtain the number of corpora; The duplicate checking threshold is analyzed based on the sample duplicate checking rate and the amount of corpus to provide a criterion for determining valid web pages.

[0011] Further, obtaining sample web pages, analyzing the sample duplication rate of the sample web pages, and extracting the number of target corpora in the sample web pages through data cleaning, and obtaining the number of corpora includes the following sub-steps: For different fields, a field keyword is obtained in each field. The field keyword is the target keyword belonging to this field. The fields are numbered and represented by the symbol U p Represents, where p∈Z+, and p is the serial number of U; Search the domain keywords through the search engine to get the search web pages, which are marked as domain web pages. The domain web pages are numbered and represented by the symbol L(p,q), where q∈Z+, and (p,q) is the sequence number of L, and L(p,q) represents U p The qth domain web page in ; Starting from p=1, L(p,q) is checked for plagiarism in pairs by permutation and combination to obtain the test plagiarism rate, which is represented by the symbol R(p,a,b), where a∈Z+, b∈Z+, and (p,a,b) is the sequence number of R, and R(p,a,b) represents the test plagiarism rate of L(p,a) and L(p,b); For any R(p,a,b), data cleaning is performed on L(p,a) and L(p,b), the same text contents in L(p,a) and L(p,b) are removed, and the average value of the amount of remaining text contents in L(p,a) and L(p,b) is marked as the corpus quantity, and each R(p,a,b) corresponds to a corpus quantity.

[0012] Furthermore, the duplicate checking threshold is analyzed based on the sample duplicate checking rate and the corpus quantity to provide a determination standard for valid web pages, including the following sub-steps: For any R(p,a,b), the amount of corpus is compared with the first quantity threshold. If the amount of corpus is less than the first quantity threshold, a signal indicating that the corpus is small is output; if the amount of corpus is greater than or equal to the first quantity threshold, a signal indicating that the corpus is sufficient is output; The average value of R(p,a,b) of the corpus with fewer signals is statistically output and marked as the duplicate checking threshold.

[0013] In a second aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the above method are performed.

[0014] In a third aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above method are performed.

[0015] Beneficial effects of the present invention: The present invention determines keywords and terms related to the target corpus, uses synonyms and related terms to expand the keyword list, searches based on the keyword list to obtain a search web page, captures the web page content of the search web page, analyzes the duplicate rate between the web page content, and extracts valid web pages in the search web page based on the duplicate rate, and then performs data cleaning on the captured web page content to remove irrelevant information and repeated information. The advantage is that there are many duplicate web pages or web pages with high duplication in the search web page. The corpus data provided by such web pages can also be extracted from other web pages. The valid corpus data that can be extracted from this web page is very small, and allocating computing resources to extract it usually wastes a lot of computing resources. Therefore, it is screened and eliminated by the duplicate rate, so that the most valid corpus can be extracted with the least computing resources when extracting the corpus data, thereby improving the effectiveness and rationality of corpus data acquisition; Before extracting valid web pages, the present invention needs to obtain sample web pages, analyze the sample duplicate rate of the sample web pages, and extract the number of target corpora in the sample web pages through data cleaning to obtain the number of corpora. Finally, the duplicate check threshold is analyzed based on the sample duplicate check rate and the number of corpora to provide a judgment standard for valid web pages. The advantage is that by analyzing the relationship between the sample duplicate check rate and the number of corpora, the duplicate check threshold is calculated to determine whether the searched web page is a valid web page, thereby improving the reliability and effectiveness of corpus data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the steps of the method of the present invention; Figure 2 A flowchart of the steps of analyzing a valid web page of the present invention; Figure 3 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1 See also Figure 1 As shown, the present application provides an AI corpus data acquisition method based on engine search, comprising the following steps: Step S1, determining keywords and terms related to the target corpus, and expanding the keyword list using synonyms and related terms; Step S1 includes the following sub-steps: Step S101, the target corpus is the corpus data to be acquired; Step S102, when it is necessary to obtain corpus data in any field, the keywords related to the field are marked as target keywords, and the terms related to the field are marked as related terms; Step S103, for the target keyword, searching for synonyms of the target keyword through a synonym library and marking them as synonymous keywords; Step S104, forming a keyword list from the target keyword, synonymous keywords and related terms; In the specific implementation, the electric power field is used as an example in this embodiment to explain the specific analysis process of corpus data acquisition. For the electric power field, a keyword library in the electric power field is obtained. The keyword library can be obtained by querying the Internet. Common target keywords in the electric power field include current, voltage, power, short circuit, power grid and other keywords. Common related terms include power load, peak and valley electricity, relay protection and other professional terms. Target keywords are usually also a kind of professional terms, but target keywords are usually more well-known, and most people know their meanings, while professional terms are usually only known by people in the field. Taking the target keyword "short circuit" as an example, the synonym of short circuit obtained by query is tripping, and the synonym is leakage. Therefore, tripping and leakage are marked as synonymous keywords. Part of the data of the keyword list composed of target keywords, synonymous keywords and related terms is shown in Table 1 below: Table 1 Partial data of keyword list

[0019] Among them, the target keyword is associated with the synonymous keyword, while the related terms have no association with the target keyword or the synonymous keyword.

[0020] See also Figure 2 As shown, step S2, based on the keyword list search to obtain the search web page, crawl the web page content of the search web page, then analyze the duplicate rate between the web page content, and extract the valid web pages in the search web page based on the duplicate rate; step S2 includes the following sub-steps: Step S201, the target keywords, synonymous keywords and related terms in the keyword list are collectively referred to as to-be-searched elements, and the to-be-searched elements are searched through different search engine APIs; Step S202, intercepting the advertisement web pages in the search results by using an advertisement blocking script, obtaining only the non-advertisement web pages in the search results, obtaining all the web pages in the search results, and marking them as search web pages; In the specific implementation, all data in the keyword list are elements to be searched. Taking relay protection as an element to be searched, the search and extraction process is specifically explained. In actual application, there are many different search engines for browsers, and the web pages searched by different search engines are different to a certain extent. Therefore, by searching for the elements to be searched through all search engines on the market, the search scope can be expanded to obtain all relevant web pages as much as possible. The ad blocking script uses the existing ad blocking technology to effectively block the advertising web pages searched out by the browser. The advertising web pages are usually web pages for promoting products, which are meaningless for the acquisition of corpus data and will occupy processing resources, so they need to be removed. By searching for "relay protection" through different search engines, 132 web pages are obtained, and the advertising web pages have been filtered by the ad blocking script. Therefore, the 132 web pages obtained by the search are the search web pages, and the first number is 132. Step S203: after obtaining the first number of search pages, number the search pages and use the symbol W n Represents, where n∈Z+, and n is the serial number of W, and Z+ represents a positive integer; Step S204, extract W n The text content in the text box is marked as T. n , set subscripts i and j, both of which are positive integers and have the same value range as n, but j≠i; Step S205, starting with i=1, using text duplication checking technology to check T i With every T j Perform duplicate analysis and obtain the duplicate rate, which is represented by the symbol C(i,j), where C(i,j) represents T j With T i The duplication rate of In the specific implementation, 132 search web pages were obtained, so W1 to W were obtained by numbering. 132 , 1≤n≤132, extract W n The text content in T n Stored in the format of a word document, 1≤i≤132, 1≤j≤132, and j≠i, taking i=1 as an example, at this time i=1, then j≠1, T j Start directly from j=2 and use it with T i Perform duplicate analysis and obtain the duplicate rate C(i,j); Step S206, analyzing C(i,j) to filter out valid web pages in the search web pages; Step S206 includes the following sub-steps: Step S2061, compare C(i,j) with the duplicate checking threshold. If C(i,j) is less than the duplicate checking threshold, output a web page pending signal; if C(i,j) is greater than or equal to the duplicate checking threshold, output a web page invalid signal; Step S2062: If the output web page is pending, temporarily retain T j If the webpage is invalid, j in C(i,j) is included in the invalid range, and the invalid range is represented by h; i+1 is added to T i With every T j Perform duplicate analysis. At this time, j≠i≠h. If i+1 is equal to any value in h, then i is increased by 1 again. Step S2063: Continue analyzing until i equals the maximum value of n, and then stop analyzing. Mark the values ​​in the non-intersection of n and h as g, and search for web page W. g Mark as a valid web page; In the specific implementation, the duplicate checking threshold needs to be calculated and analyzed by step S4. The analysis of the duplicate checking threshold is displayed at the end of this embodiment to prevent repeated description when describing the duplicate checking rate and data cleaning in the duplicate checking threshold analysis process. The origin of the duplicate checking rate and the processing process of data cleaning are displayed by steps S2 and S3, and step S4 can be more clean when cited; taking C(2,4) as an example, C(2,4) represents the duplicate checking rate of T4 and T2. Through analysis, it is obtained that C(2,4) is greater than the duplicate checking threshold, and an invalid signal of the web page is output. At this time, j=4, that is, 4 is included in the invalid range h. When analyzing at i=3, there is no need to calculate C(3,4), because T4 has been eliminated. This embodiment only expresses the need to eliminate T j , when actually removing, you need to refer to T i and T j The length of the text, that is, the number of words, usually retains the web page with more words, for example, C(2,4) in this embodiment, C(2,4) represents the duplicate check rate of T4 and T2, and outputs an invalid web page signal. By comparing the number of words of T2 and T4, it is found that the number of words in T2 is less than that of T4, then T2 needs to be eliminated, that is, 2 is included in the invalid range h instead of 4. At this time, there is no need to calculate the case of i=2, and i+1 is directly used for the next analysis; continue to analyze until i equals the maximum value of n and stop the analysis. At this time, 1≤n≤132, and h includes different numbers. After eliminating the numbers in h within the range of n, the remaining numbers are g, W g This is a valid web page.

[0021] Step S3, cleaning the captured webpage content to remove irrelevant and duplicate information; Step S3 includes the following sub-steps: Step S301, in the front-end design of a web page, a web page is usually divided into a title part, a navigation part and a body part, and the body part of a valid web page is extracted; Step S302: Mark the text content between the first line indentation format and the line break in the body part as the body content, and number the body content, represented by the symbol F(g,d), where d∈Z+, and (g,d) is the sequence number of F, and F(g,d) represents the valid web page W g The content of Article d in the text; Step S303, counting all F(g, d), and comparing the contents of each F(g, d). If there are F(g, d) with completely identical contents, one of the main text contents is removed, and the removed main text content is no longer involved in the content comparison; Step S304: After the content comparison and analysis is completed, the remaining text content is marked as target corpus and stored in the AI ​​corpus; In the specific implementation, in the front-end design of the web page, the title part, the navigation part and the body part usually have certain identification codes, which can be directly extracted. The text content between the first line indentation format and the line break in the body part is marked as the body content, that is, the body part is extracted according to the paragraph; when performing content comparison, valid web pages are not distinguished, F(g,d) is regarded as a set, and all elements in this set are compared with each other. Assuming that there are 1000 F(g,d) in total, each F(g,d) needs to be compared 999 times. Every time an F(g,d) is eliminated, one less comparison can be performed. When the content is detected When there are exactly the same F(g, d), just randomly remove one of the two F(g, d), because they are exactly the same. Although there will be synonymous paragraphs when extracting corpus data, that is, the same meaning, two expressions, they are usually removed in other application fields. For example, in the field of data alignment, synonymous duplicate data also need to be removed, but the corpus data is used for the training of AI large language models, so this kind of multi-expression sentences also need to be recorded to broaden the training samples, so only the exactly the same F(g, d) needs to be removed; after the content comparison analysis is completed, the remaining text content is marked as the target corpus and stored in the AI ​​corpus.

[0022] Step S4, setting up a duplicate checking test experiment, analyzing the sample duplicate checking rate through sample web pages, and combining data cleaning analysis to obtain a duplicate checking threshold to provide a criterion for determining valid web pages; Step S4 includes the following sub-steps: Step S401, obtaining a sample web page, analyzing the sample duplication rate of the sample web page, and extracting the number of target corpora in the sample web page through data cleaning to obtain the number of corpora; Step S401 includes the following sub-steps: Step S4011, for different fields, a field keyword is obtained in each field, and the field keyword is the target keyword belonging to this field, and the fields are numbered and represented by the symbol U p Represents, where p∈Z+, and p is the serial number of U; Step S4012, search the domain keyword through the search engine, obtain the search web page, mark it as the domain web page, number the domain web page, and represent it by the symbol L(p,q), where q∈Z+, and (p,q) is the sequence number of L, L(p,q) represents U p The qth domain web page in ; In the specific implementation, for different fields, such as the power field, the agricultural field, the oil and gas extraction field, and the food manufacturing field, a field keyword is randomly extracted from each field for analyzing the duplicate check threshold; in this embodiment, 263 fields are subdivided, namely U1 to U 263 , 1≤p≤263, for U p The step of extracting the domain webpage is the same as the step of extracting the search webpage in step S2, so it will not be specifically demonstrated in this embodiment. It is worth noting that the domain webpage is the search webpage in step S2, not the valid webpage; Step S4013, starting with p=1, perform duplication analysis on L(p,q) in pairs by permutation and combination to obtain a test duplication rate, represented by the symbol R(p,a,b), where a∈Z+, b∈Z+, and (p,a,b) is the sequence number of R, and R(p,a,b) represents the test duplication rate of L(p,a) and L(p,b); Step S4014: for any R(p,a,b), perform data cleaning on L(p,a) and L(p,b), remove the same text contents in L(p,a) and L(p,b), and mark the average of the amount of remaining text contents in L(p,a) and L(p,b) as the corpus quantity. Each R(p,a,b) corresponds to a corpus quantity. In the specific implementation, when p=1, only L(1,q) is analyzed in pairs to obtain R(p,a,b), that is, the domain web pages in the same domain are analyzed with each other. After analyzing all the test duplication rates, all duplicate data in the two domain web pages are removed by data cleaning. The retained data is the valid corpus data that can be extracted from the two domain web pages when the duplication rate is the test duplication rate. Then, their average value is taken as the number of corpora for this group of test duplication rates. Step S402, analyzing the duplicate checking threshold based on the sample duplicate checking rate and the corpus quantity to provide a criterion for determining a valid web page; Step S402 includes the following sub-steps: Step S4021, for any R(p,a,b), compare the amount of corpus with the first amount threshold, if the amount of corpus is less than the first amount threshold, output a signal indicating that the corpus is small; if the amount of corpus is greater than or equal to the first amount threshold, output a signal indicating that the corpus is sufficient; Step S4022, the average value of R(p, a, b) of the signal with less corpus is counted and output, and marked as the duplicate checking threshold; In a specific implementation, the first quantity threshold can be set by oneself. After long-term experiments, it is proved that the first quantity threshold is best set between 5 and 20. The smaller the first quantity threshold is, the more complicated the processing process is and the longer the processing time is, but the more valid corpus is extracted. The larger the first quantity threshold is, the simpler the processing process is and the shorter the processing time is, but the less valid corpus is extracted. It can be determined by the user. In this embodiment, the first quantity threshold is set to 10. For example, R(3,6,9)=84.73% in this embodiment, and the corresponding corpus quantity is 9. By comparison, it is found that the corpus quantity is less than the first quantity threshold, and a corpus less signal is output. The average value of R(p,a,b) of all output corpus less signals is calculated, which is the duplicate checking threshold. In this embodiment, the duplicate checking threshold is calculated to be 82.64%, that is, the search web pages with a duplicate checking rate higher than 82.64% can eliminate less than 10 valid corpus data, and the extraction of them is of little significance, so they are directly omitted.

[0023] Example 2 See also Figure 3 As shown, Figure 3 The schematic diagram of the structure of an electronic device is illustrated, and the electronic device may include: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The memory stores computer-readable instructions, and the processor can call the instructions in the memory. When the computer-readable instructions are executed by the processor, the steps in the AI ​​corpus data acquisition method based on engine search are executed to achieve the following functions: determine the keywords and terms related to the target corpus, and use synonyms and related terms to expand the keyword list; search based on the keyword list to obtain the search web page, and extract the valid web pages in the search web page based on the duplicate check rate; perform data cleaning on the captured web page content to remove irrelevant information and repeated information; set up a duplicate check test experiment, analyze the sample duplicate check rate through the sample web page, and obtain the duplicate check threshold value in combination with data cleaning analysis.

[0024] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0025] Example 3 The present application also provides a computer-readable storage medium. The present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned AI corpus data acquisition method based on engine search are executed to achieve the following functions: determine keywords and terms related to the target corpus, and expand the keyword list using synonyms and related terms; obtain search web pages based on the keyword list search, and extract valid web pages from the search web pages based on the duplicate check rate; perform data cleaning on the captured web page content to remove irrelevant information and duplicate information; set up a duplicate check test experiment, analyze the sample duplicate check rate through sample web pages, and obtain the duplicate check threshold in combination with data cleaning analysis.

[0026] Through the description of the above implementation methods, the embodiments of the present invention can be provided as methods, systems or computer program products. Based on such an understanding, the above technical solutions can be essentially or partly contributed to the prior art in the form of software products, which can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and include several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0027] In the embodiments provided in the present application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of systems, modules and units can be electrical, mechanical or other forms.

[0028] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. The AI ​​corpus data acquisition method based on engine search is characterized by: The steps include: Identify keywords and terms relevant to the target corpus and expand the keyword list using synonyms and related terms; Based on the keyword list, a search webpage is obtained, the webpage content of the search webpage is crawled, the duplicate rate between the webpage contents is analyzed, and valid webpages in the search webpage are extracted based on the duplicate rate; Clean the captured webpage content to remove irrelevant and duplicate information; Set up a duplicate checking test experiment, analyze the sample duplicate checking rate through sample web pages, and combine data cleaning analysis to obtain the duplicate checking threshold to provide a judgment standard for valid web pages.

2. The method for acquiring AI corpus data based on engine search according to claim 1, characterized in that: Identify keywords and terms related to the target corpus and expand the keyword list using synonyms and related terms. This includes the following sub-steps: The target corpus is the corpus data that needs to be acquired; When you need to obtain corpus data in any field, mark the keywords related to this field as target keywords, and mark the terms related to this field as related terms; For the target keyword, search for synonyms of the target keyword through the thesaurus and mark them as synonymous keywords; The keyword list consists of the target keyword, synonymous keywords, and related terms.

3. The method for acquiring AI corpus data based on engine search according to claim 2, characterized in that: The search webpage is obtained based on the keyword list search, the webpage content of the search webpage is crawled, the duplicate rate between the webpage contents is analyzed, and the valid webpages in the search webpage are extracted based on the duplicate rate, including the following sub-steps: The target keywords, synonymous keywords and related terms in the keyword list are collectively referred to as search elements, and the search elements are searched through different search engine APIs; Advertisement pages in search results are blocked by ad blocking scripts, only non-advertisement pages in search results are obtained, all pages in search results are obtained, and marked as search pages; After obtaining the first number of search pages, the search pages are numbered and represented by symbols W. n Indicates, where n∈Z+, and n is the serial number of W, and Z+ represents a positive integer; Extract W n The text content in the text box is marked as T. n , set subscripts i and j, where both i and j are positive integers and the value range of i and j is the same as that of n, but j≠i; Starting from i=1, T i With every T j Perform duplicate analysis and obtain the duplicate rate, which is represented by the symbol C(i,j), where C(i,j) represents T j With T i The duplication rate of Analyze C(i,j) and filter out valid web pages in the search web pages.

4. The method for acquiring AI corpus data based on engine search according to claim 3, characterized in that: Analyzing C(i,j) and screening valid web pages in the search web pages includes the following sub-steps: Compare C(i,j) with the duplicate checking threshold. If C(i,j) is less than the duplicate checking threshold, output a pending signal for the webpage; if C(i,j) is greater than or equal to the duplicate checking threshold, output an invalid signal for the webpage. If the output web page is pending, temporarily retain T j If the webpage is invalid, j in C(i,j) is included in the invalid range, and the invalid range is represented by h; i+1 is added to T i With every T j Perform duplicate analysis. At this time, j≠i≠h. If i+1 is equal to any value in h, then i is increased by 1 again. Continue analyzing until i equals the maximum value of n, and then mark the values ​​in the non-intersection of n and h as g, and search for web page W. g Mark as a valid page.

5. The method for acquiring AI corpus data based on engine search according to claim 4, characterized in that: Data cleaning of the captured webpage content to remove irrelevant and duplicate information includes the following sub-steps: In the front-end design of web pages, web pages are usually divided into the title part, the navigation part and the text part. The text part of the effective web page is extracted; Mark the text content between the first line indentation format and the line break in the body as the body content, and number the body content, represented by the symbol F(g,d), where d∈Z+, and (g,d) is the serial number of F, and F(g,d) represents the valid web page W g The content of Article d in the text; Count all F(g,d), and compare the content of each F(g,d). If there are F(g,d) with exactly the same content, remove the content of one of them, and the removed content will no longer participate in the content comparison. After the content comparison and analysis is completed, the remaining text content is marked as target corpus and stored in the AI ​​corpus.

6. The method for acquiring AI corpus data based on engine search according to claim 5, characterized in that: Setting up a duplicate checking test experiment, analyzing the sample duplicate checking rate through sample web pages, and combining data cleaning analysis to obtain the duplicate checking threshold, so as to provide the judgment criteria for valid web pages, includes the following sub-steps: Obtain sample web pages, analyze the sample duplication rate of the sample web pages, and extract the number of target corpora in the sample web pages through data cleaning to obtain the number of corpora; The duplicate checking threshold is analyzed based on the sample duplicate checking rate and the amount of corpus to provide a criterion for determining valid web pages.

7. The method for acquiring AI corpus data based on engine search according to claim 6, characterized in that: Obtaining sample web pages, analyzing the sample duplication rate of the sample web pages, and extracting the number of target corpora in the sample web pages through data cleaning, and obtaining the number of corpora includes the following sub-steps: For different fields, a field keyword is obtained in each field. The field keyword is the target keyword belonging to this field. The fields are numbered and represented by the symbol U p Represents, where p∈Z+, and p is the serial number of U; Search the domain keywords through the search engine to get the search web pages, which are marked as domain web pages. The domain web pages are numbered and represented by the symbol L(p,q), where q∈Z+, and (p,q) is the sequence number of L, and L(p,q) represents U p The qth domain web page in ; Starting from p=1, L(p,q) is checked for plagiarism in pairs by permutation and combination to obtain the test plagiarism rate, which is represented by the symbol R(p,a,b), where a∈Z+, b∈Z+, and (p,a,b) is the sequence number of R, and R(p,a,b) represents the test plagiarism rate of L(p,a) and L(p,b); For any R(p,a,b), data cleaning is performed on L(p,a) and L(p,b), the same text contents in L(p,a) and L(p,b) are removed, and the average value of the amount of remaining text contents in L(p,a) and L(p,b) is marked as the corpus quantity, and each R(p,a,b) corresponds to a corpus quantity.

8. The method for acquiring AI corpus data based on engine search according to claim 7, characterized in that: Analyzing the duplicate checking threshold based on the sample duplicate checking rate and the amount of corpus to provide the criteria for determining valid web pages includes the following sub-steps: For any R(p,a,b), the amount of corpus is compared with the first amount threshold, and if the amount of corpus is less than the first amount threshold, a signal indicating that the corpus is small is output; If the amount of corpus is greater than or equal to the first amount threshold, outputting a corpus sufficiency signal; The average value of R(p,a,b) of the corpus with fewer signals is statistically output and marked as the duplicate checking threshold.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 8 are executed.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are executed.

Images