Financial information crawling method based on web crawler

Abstract

The invention relates to the technical field of web crawlers, and discloses a web crawler-based financial information crawling method, which comprises the following steps of: collecting websites needing to be crawled, performing crawling test on the websites, judging whether the current website can be crawled or not according to a crawling test result, and crawling the websites capable of being crawled. The data conforming to the keyword is used as a first set, a crawling data collection strategy is carried out on the first set, and the data collection strategy specifically comprises the steps that the number of data collected each time is set, elements in the first set are selected, a comparison set is input, and the number of repetitions between the data collected each time and the elements in the comparison set is judged; and the repetition rate of each time is calculated, and the number of times of subsequent crawling is predicted according to the repetition rate.

Description

Technical Field

[0001] The present invention relates to the technical field of web crawlers, and in particular to a financial information crawling method based on web crawlers. Background Art

[0002] The origin of web crawlers is mainly due to the demand for massive data in the big data era. With the rapid development of the Internet, the World Wide Web has become a carrier of a large amount of information. How to use this information quickly and correctly has become a challenge. Web crawler technology came into being and became a program or script that automatically extracts web page information. It automatically crawls information on the World Wide Web according to certain rules. This technology not only improves the efficiency of information collection, but also reduces labor costs, making information acquisition more convenient.

[0003] The research and development background of web crawlers can be traced back to the emergence of search engines. Early search engines needed to extract useful information from a large number of web pages, but manual work was not only inefficient but also costly. Therefore, it became necessary to develop programs that could automatically crawl, process, and index web page content. These programs were originally called web crawlers or web spiders, which could simulate human browsing behavior, crawl data from the Internet, and provide data support for search engines.

[0004] The technical implementation of web crawlers is not limited to a specific programming language, but can be developed using a variety of programming languages, including but not limited to Python, Java, PHP, C / C++ / C#, Go, and Node.js, etc. These languages ​​can be used to write crawler programs or scripts to achieve automated information capture and processing.

[0005] In general, web crawler technology is a very important tool that can help us better obtain the data we want. However, since the data obtained by crawling technology is too large and there is data redundancy, it leads to crawling too much unnecessary data, increasing the workload of information screening.

[0006] In summary, there is an urgent need for a crawling method based on a web crawler to solve the above problems. Summary of the invention

[0007] The present invention provides a financial information crawling method based on a web crawler, which is used to promote the solution of the problems mentioned in the above background technology.

[0008] The present invention provides the following technical solution: a financial information crawling method based on a web crawler,

[0009] Optionally, a financial information crawling method based on a web crawler includes:

[0010] Collect website information and perform crawling tests on the website;

[0011] Determine whether to perform crawling operations based on the crawling test results;

[0012] If the crawling operation is not performed, reselect the URL and perform a crawling test;

[0013] If a crawling operation is performed, the URL is pre-processed for crawling;

[0014] Take the crawl preprocessing results as the first set;

[0015] Crawling data collection strategy for the first set;

[0016] The crawling data collection strategy includes the first crawling operation, the second crawling operation and the third crawling operation;

[0017] Calculate crawl repetition rate through crawl data collection strategy;

[0018] According to the crawling repetition rate prediction, the estimated number of crawling data processing times;

[0019] According to the estimated number of crawled data processing times, the first round of predicted crawled data processing is carried out;

[0020] According to the results of the first round of prediction crawling data processing, determine whether to perform the second round of prediction crawling data processing;

[0021] If the second round of prediction crawling data processing is not performed, the results of the first round of prediction crawling data processing are retained;

[0022] If a second round of prediction crawling data processing is performed, the results of the second round of prediction crawling data processing will be retained.

[0023] Optionally, the collecting of website information and crawling test of the website include:

[0024] Collect all financial information URLs and enter them into a crawled URL set;

[0025] Select the elements in the crawled URL set as the target URL in sequence;

[0026] Perform crawling test on the target website;

[0027] If the number of data obtained through the crawling test is zero, then reselect an element in the crawled URL set as a new target website;

[0028] If the number of data obtained through the crawling test is not zero, the target website is crawled and preprocessed.

[0029] Optionally, if the crawling operation is performed, the URL is crawled and pre-processed, including:

[0030] Set the first crawling interval and target keywords;

[0031] Crawling all crawled data including the target keyword within the first crawling interval, and recording the crawled data into the first set;

[0032] Set data volume threshold;

[0033] Get the number of elements in the first set;

[0034] If the number of elements in the first set is less than the data volume threshold, set the second crawling interval;

[0035] crawling all crawling data including the target keyword in a second crawling interval, where the second crawling interval includes the first crawling interval;

[0036] If the number of elements in the first set is greater than or equal to the data volume threshold, a crawling data collection strategy is performed on the elements in the first set.

[0037] Optionally, the crawling data collection strategy for the first set includes:

[0038] Set the crawl quantity threshold a;

[0039] Perform the first crawl operation;

[0040] Randomly select a number of crawled data in the first set and enter them into the comparison set.

[0041] Optionally, the crawling data collection strategy for the first set includes:

[0042] Perform a second crawl operation;

[0043] Obtain the intersection between the first set and the comparison set, recorded as the first intersection set;

[0044] Eliminate all first intersection set elements in the first set;

[0045] Randomly select crawled data with a number equal to crawled data threshold a in the first set, and enter them into the first pre-selected set;

[0046] In the first pre-selected set, each element is selected in turn, and each selected element is compared with all elements in the comparison set by using natural language processing technology;

[0047] If the comparison result is identical, the currently selected element is marked as a duplicate element;

[0048] If the comparison result is non-identical, the currently selected element is entered into the comparison set;

[0049] Get the number of repeated elements;

[0050] Calculate the ratio between the number of repeated elements and the crawled data threshold a, and record it as the first repetition rate.

[0051] Optionally, the crawling data collection strategy for the first set includes:

[0052] Perform the third crawl operation;

[0053] Obtain the intersection between the first set and the comparison set, recorded as the second intersection set;

[0054] Eliminate all elements of the second intersection set in the first set;

[0055] Randomly select crawled data with a number equal to crawled data threshold a in the first set, and enter them into the second pre-selected set;

[0056] In the second pre-selected set, each element is selected in turn, and each selected element is compared with all elements in the comparison set by using natural language processing technology;

[0057] If the comparison result is identical, then mark the current element in the second pre-selected set as a duplicate element;

[0058] If the comparison result is non-identical, the current element in the second pre-selected set is entered into the comparison set;

[0059] Get the number of repeated elements;

[0060] Calculate the ratio between the number of repeated elements and the crawled data threshold a, and record it as the second repetition rate;

[0061] By calculating the first repetition rate and the second repetition rate, the number of crawling times is predicted.

[0062] Optionally, the predicting the number of crawling times by calculating the first repetition rate and the second repetition rate includes:

[0063] Entering the first repetition rate and the second repetition rate into a repetition set;

[0064] Calculate the difference between adjacent elements in the repeating set, denoted as b1, b2...bn, where n is the number of elements in the repeating set;

[0065] The first average repetition rate is calculated by the following formula:

[0066]

[0067] In the above formula, d represents the first average repetition rate, bi represents the i-th difference between adjacent elements in the repetition set, i represents the number of elements in the current repetition set, and n represents the number of elements in the repetition set;

[0068] Set the crawling cutoff threshold e;

[0069] The estimated number of crawling data processing times is calculated using the following formula:

[0070]

[0071] In the above formula, f represents the expected number of crawled data processing times, c represents the number of elements in the repeated set, bn represents the nth element in the repeated set, n is the number of elements in the repeated set, e represents the crawling cutoff threshold, d represents the first average repetition rate, Indicates rounding up.

[0072] Optionally, performing a first round of prediction crawling data processing according to the predicted number of crawling times required includes:

[0073] The first set is crawled f times, and the steps are as follows:

[0074] S1 obtains the intersection between the first set and the comparison set, and removes all elements of the intersection in the first set;

[0075] S2 randomly selects crawled data whose number in the first set is the crawled data threshold a, and enters them into the pre-selected set;

[0076] S3 selects each element in the first pre-selected set in turn by using natural language processing technology, and compares the selected element each time with all elements in the comparison set;

[0077] S4 If the comparison result is identical, then mark the current element in the pre-selected set as a duplicate element;

[0078] S5: if the comparison result is non-identical, the current element in the pre-selected set is entered into the comparison set;

[0079] S6 obtains the number of repeated elements, calculates the ratio between the number of repeated elements and the crawled data threshold a, which is the repetition rate of the current crawled data processing, and enters the repetition rate of the current crawled data processing into the repetition set;

[0080] S7 records the current crawling processing times, recorded as g;

[0081] If g is less than f, repeat the above steps S1-S7;

[0082] If g is equal to f, the crawling operation ends;

[0083] Sort the elements in the repeating set in the order of entry, obtain the number of elements in the repeating set, record it as m, select the mth element in the repeating set, record it as the comparison element;

[0084] If the comparison element is less than or equal to the crawling cutoff threshold e, the crawling process is stopped and all elements in the comparison set are retained;

[0085] If the comparison element is greater than the crawling cutoff threshold e, a second estimated crawling data processing is performed.

[0086] Optionally, if a second round of prediction crawling data processing is required, the second round of prediction crawling data processing result is fed back to the user, including:

[0087] The second average repetition rate is calculated by the following formula:

[0088]

[0089] In the above formula, r represents the second average repetition rate, bq represents the qth difference between adjacent elements in the repetition set, q represents the number of elements in the current repetition set, and p represents the number of elements in the repetition set;

[0090] The second estimated crawling data processing times are calculated using the following formula:

[0091]

[0092] In the above formula, h represents the second estimated number of crawling data processing times, n represents the number of elements in the repeated set, bp represents the pth element in the repeated set, e represents the crawling cutoff threshold, and r represents the second average repetition rate. Indicates rounding up;

[0093] Perform h crawling operations on the first set;

[0094] S1 removes all elements in the comparison set from the first set;

[0095] S2 randomly selects crawled data whose number in the first set is the crawled data threshold a, and enters them into the pre-selected set;

[0096] S3 uses natural language processing technology to sequentially judge the elements in the pre-selected set and all the elements in the comparison set;

[0097] If the judgment result is identical, then the current element in the pre-selected set is marked as a duplicate element;

[0098] If the judgment result is non-identical, then the current element in the pre-selected set is entered into the comparison set;

[0099] S6 obtains the number of repeated elements, calculates the ratio between the number of repeated elements and the crawling data threshold a, which is the repetition rate of the current crawling process, and enters the repetition rate of the current crawling process into the repetition set;

[0100] S7 records the current crawling processing times, recorded as k;

[0101] If k is less than h, repeat the above steps S1 to S7;

[0102] If k is equal to h, the crawling process is stopped and all elements in the comparison set are retained.

[0103] The present invention has the following beneficial effects:

[0104] 1. This financial information crawling method based on web crawlers collects relevant URL information and performs crawling tests on the URL information to ensure the relevance of the crawled data with the target data. At the same time, it ensures that the collected URLs can be used for data crawling, thereby ensuring the feasibility of the crawling method.

[0105] 2. This financial information crawling method based on web crawlers sets a data volume threshold to determine whether the data of the URL within the crawling time range meets the crawling requirements. If not, the crawling data can be increased by adjusting the crawling time range. In this way, crawling adjustments can be made according to different needs, which is flexible and universal.

[0106] 3. This financial information crawling method based on a web crawler performs multiple crawling operations on the data in a first set, enters the elements obtained in each crawling operation into a comparison set, and then compares the data of each crawling operation with the elements in the comparison set to calculate the repetition rate of each crawling operation. By using this method, the repetition between each crawling result and the previous crawling result is digitized, so that the efficiency of each crawling can be more intuitively understood.

[0107] 4. A financial information crawling method based on a web crawler obtains the repetition rate of each crawling operation and sums them up, then calculates the average repetition rate through a formula, sets a crawling cutoff threshold, and then calculates the subsequent expected number of crawling operations based on the average repetition rate and the crawling cutoff threshold. According to the expected number of crawling operations, a first round of predictive crawling data processing is performed. If the repetition rate of the first round of crawling data processing results is less than the crawling cutoff threshold, all elements in the current comparison set are retained. If the repetition rate of the first round of crawling data processing results is greater than or equal to the crawling cutoff threshold, a second round of predictive crawling data processing is performed and all elements in the current comparison set are retained. This method can predict the number of crawling times required in the future, thereby optimizing the crawling process and reducing unnecessary resource consumption. Moreover, according to the predicted number of crawling times and repetition rate, the method can dynamically adjust the crawling process to ensure that while meeting data needs, excessive crawling is avoided. Through natural language processing technology and repetition rate calculation, duplicate data can be effectively identified and eliminated to improve the uniqueness and value of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 It is a schematic diagram of the structure of the present invention. DETAILED DESCRIPTION

[0109] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0110] Example 1, see Figure 1 ,A financial information crawling method based on a web crawler, comprising,

[0111] Collect website information and perform crawling tests on the website;

[0112] Determine whether to perform crawling operations based on the crawling test results;

[0113] If the crawling operation is not performed, reselect the URL and perform a crawling test;

[0114] If a crawling operation is performed, the URL is pre-processed for crawling;

[0115] Take the crawl preprocessing results as the first set;

[0116] Crawling data collection strategy for the first set;

[0117] The crawling data collection strategy includes the first crawling operation, the second crawling operation and the third crawling operation;

[0118] Calculate crawl repetition rate through crawl data collection strategy;

[0119] According to the crawling repetition rate prediction, the estimated number of crawling data processing times;

[0120] According to the estimated number of crawled data processing times, the first round of predicted crawled data processing is carried out;

[0121] According to the results of the first round of prediction crawling data processing, determine whether to perform the second round of prediction crawling data processing;

[0122] If the second round of prediction crawling data processing is not performed, the results of the first round of prediction crawling data processing are retained;

[0123] If a second round of prediction crawling data processing is performed, the results of the second round of prediction crawling data processing will be retained.

[0124] The collecting of website information and crawling test of the website include:

[0125] Collect all financial information URLs and enter them into a crawled URL set;

[0126] Select the elements in the crawled URL set as the target URL in sequence;

[0127] Perform crawling test on the target website;

[0128] If the number of data obtained through the crawling test is zero, then reselect an element in the crawled URL set as a new target website;

[0129] If the number of data obtained through the crawling test is not zero, the target website is crawled and preprocessed.

[0130] If the crawling operation is performed, the URL is crawled and pre-processed, including:

[0131] Set the first crawling interval and target keywords;

[0132] The first crawling interval is the time range for crawling, which is used to limit the time area for crawling;

[0133] The target keyword is the crawled data required by the user, which is used to filter out other unnecessary data;

[0134] Crawling all crawled data including the target keyword within the first crawling interval, and recording the crawled data into the first set;

[0135] Set data volume threshold;

[0136] The data volume threshold is a constant and can be adjusted according to website results and user needs. It is used to determine whether the data volume in the current website meets the crawling requirements.

[0137] Get the number of elements in the first set;

[0138] If the number of elements in the first set is less than the data volume threshold, set the second crawling interval;

[0139] crawling all crawling data including the target keyword in a second crawling interval, where the second crawling interval includes the first crawling interval;

[0140] If the number of elements in the first set is greater than or equal to the data volume threshold, a crawling data collection strategy is performed on the elements in the first set.

[0141] The crawling data collection strategy for the first set includes:

[0142] Set the crawl quantity threshold a;

[0143] The crawl quantity threshold a is a constant and can be adjusted according to the user. It is used to limit the amount of data crawled in a single time and calculate the repetition rate of a single crawl.

[0144] Perform the first crawl operation;

[0145] Randomly select a number of crawled data in the first set and enter them into the comparison set.

[0146] The crawling data collection strategy for the first set includes:

[0147] Perform a second crawl operation;

[0148] Obtain the intersection between the first set and the comparison set, recorded as the first intersection set;

[0149] Eliminate all first intersection set elements in the first set;

[0150] Randomly select crawled data with a number equal to crawled data threshold a in the first set, and enter them into the first pre-selected set;

[0151] In the first pre-selected set, each element is selected in turn, and each selected element is compared with all elements in the comparison set by using natural language processing technology;

[0152] If the comparison result is identical, the currently selected element is marked as a duplicate element;

[0153] If the comparison result is non-identical, the currently selected element is entered into the comparison set;

[0154] Get the number of repeated elements;

[0155] Calculate the ratio between the number of repeated elements and the crawled data threshold a, and record it as the first repetition rate.

[0156] The crawling data collection strategy for the first set includes:

[0157] Perform the third crawl operation;

[0158] Obtain the intersection between the first set and the comparison set, recorded as the second intersection set;

[0159] Eliminate all elements of the second intersection set in the first set;

[0160] Randomly select crawled data with a number equal to crawled data threshold a in the first set, and enter them into the second pre-selected set;

[0161] In the second pre-selected set, each element is selected in turn, and each selected element is compared with all elements in the comparison set by using natural language processing technology;

[0162] If the comparison result is identical, then mark the current element in the second pre-selected set as a duplicate element;

[0163] If the comparison result is non-identical, the current element in the second pre-selected set is entered into the comparison set;

[0164] Get the number of repeated elements;

[0165] Calculate the ratio between the number of repeated elements and the crawled data threshold a, and record it as the second repetition rate;

[0166] By calculating the first repetition rate and the second repetition rate, the number of crawling times is predicted.

[0167] The method of predicting the number of crawling times by calculating the first repetition rate and the second repetition rate includes:

[0168] Entering the first repetition rate and the second repetition rate into a repetition set;

[0169] Calculate the difference between adjacent elements in the repeating set, denoted as b1, b2...bn, where n is the number of elements in the repeating set;

[0170] The first average repetition rate is calculated by the following formula:

[0171]

[0172] In the above formula, d represents the first average repetition rate, bi represents the i-th difference between adjacent elements in the repetition set, i represents the number of elements in the current repetition set, and n represents the number of elements in the repetition set;

[0173] Set the crawling cutoff threshold e;

[0174] The crawling cutoff threshold is a constant and can be adjusted according to user needs to determine whether to stop the crawling operation;

[0175] The estimated number of crawling data processing times is calculated using the following formula:

[0176]

[0177] In the above formula, f represents the expected number of crawled data processing times, c represents the number of elements in the repeated set, bn represents the nth element in the repeated set, n is the number of elements in the repeated set, e represents the crawling cutoff threshold, d represents the first average repetition rate, Indicates rounding up.

[0178] The first round of prediction crawling data processing is performed according to the predicted number of crawling times required, including:

[0179] The first set is crawled f times, and the steps are as follows:

[0180] S1 obtains the intersection between the first set and the comparison set, and removes all elements of the intersection in the first set;

[0181] S2 randomly selects crawled data whose number in the first set is the crawled data threshold a, and enters them into the pre-selected set;

[0182] S3 selects each element in the first pre-selected set in turn by using natural language processing technology, and compares the selected element each time with all elements in the comparison set;

[0183] S4 If the comparison result is identical, then mark the current element in the pre-selected set as a duplicate element;

[0184] S5: if the comparison result is non-identical, the current element in the pre-selected set is entered into the comparison set;

[0185] S6 obtains the number of repeated elements, calculates the ratio between the number of repeated elements and the crawled data threshold a, which is the repetition rate of the current crawled data processing, and enters the repetition rate of the current crawled data processing into the repetition set;

[0186] S7 records the current crawling processing times, recorded as g;

[0187] If g is less than f, repeat the above steps S1-S7;

[0188] If g is equal to f, the crawling operation ends;

[0189] Sort the elements in the repeating set in the order of entry, obtain the number of elements in the repeating set, record it as m, select the mth element in the repeating set, record it as the comparison element;

[0190] If the comparison element is less than or equal to the crawling cutoff threshold e, the crawling process is stopped and all elements in the comparison set are retained;

[0191] If the comparison element is greater than the crawling cutoff threshold e, a second estimated crawling data processing is performed.

[0192] If a second round of prediction crawling data processing is required, the second round of prediction crawling data processing results are fed back to the user, including:

[0193] The second average repetition rate is calculated by the following formula:

[0194]

[0195] In the above formula, r represents the second average repetition rate, bq represents the qth difference between adjacent elements in the repetition set, q represents the number of elements in the current repetition set, and p represents the number of elements in the repetition set;

[0196] The second estimated crawling data processing times are calculated using the following formula:

[0197]

[0198] In the above formula, h represents the second estimated number of crawling data processing times, n represents the number of elements in the repeated set, bp represents the pth element in the repeated set, e represents the crawling cutoff threshold, and r represents the second average repetition rate. Indicates rounding up;

[0199] Perform h crawling operations on the first set;

[0200] S1 removes all elements in the comparison set from the first set;

[0201] S2 randomly selects crawled data whose number in the first set is the crawled data threshold a, and enters them into the pre-selected set;

[0202] S3 uses natural language processing technology to sequentially judge the elements in the pre-selected set and all the elements in the comparison set;

[0203] If the judgment result is identical, then the current element in the pre-selected set is marked as a duplicate element;

[0204] If the judgment result is non-identical, then the current element in the pre-selected set is entered into the comparison set;

[0205] S6 obtains the number of repeated elements, calculates the ratio between the number of repeated elements and the crawling data threshold a, which is the repetition rate of the current crawling process, and enters the repetition rate of the current crawling process into the repetition set;

[0206] S7 records the current crawling processing times, recorded as k;

[0207] If k is less than h, repeat the above steps S1 to S7;

[0208] If k is equal to h, the crawling process is stopped and all elements in the comparison set are retained.

[0209] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0210] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A financial information crawling method based on a web crawler, characterized in that: include, Collect website information and perform crawling tests on the website; Determine whether to perform crawling operations based on the crawling test results; If the crawling operation is not performed, reselect the URL and perform a crawling test; If a crawling operation is performed, the URL is pre-processed for crawling; Take the crawl preprocessing results as the first set; Crawling data collection strategy for the first set; The crawling data collection strategy includes the first crawling operation, the second crawling operation and the third crawling operation; Calculate crawl repetition rate through crawl data collection strategy; According to the crawling repetition rate prediction, the estimated number of crawling data processing times; According to the estimated number of crawled data processing times, the first round of predicted crawled data processing is carried out; According to the results of the first round of prediction crawling data processing, determine whether to perform the second round of prediction crawling data processing; If the second round of prediction crawling data processing is not performed, the results of the first round of prediction crawling data processing are retained; If a second round of prediction crawling data processing is performed, the results of the second round of prediction crawling data processing will be retained.

2. The method for crawling financial information based on a web crawler according to claim 1, characterized in that: The collecting of website information and crawling test of the website include: Collect all financial information URLs and enter them into a crawled URL set; Select the elements in the crawled URL set as the target URL in sequence; Perform crawling test on the target website; If the number of data obtained through the crawling test is zero, then reselect an element in the crawled URL set as a new target website; If the number of data obtained through the crawling test is not zero, the target website is crawled and preprocessed.

3. The method for crawling financial information based on a web crawler according to claim 1, characterized in that: If the crawling operation is performed, the URL is crawled and pre-processed, including: Set the first crawling interval and target keywords; Crawling all crawled data including the target keyword within the first crawling interval, and recording the crawled data into the first set; Set data volume threshold; Get the number of elements in the first set; If the number of elements in the first set is less than the data volume threshold, set the second crawling interval; crawling all crawling data including the target keyword in a second crawling interval, where the second crawling interval includes the first crawling interval; If the number of elements in the first set is greater than or equal to the data volume threshold, a crawling data collection strategy is performed on the elements in the first set.

4. The method for crawling financial information based on a web crawler according to claim 1, characterized in that: The crawling data collection strategy for the first set includes: Set the crawl quantity threshold a; Perform the first crawl operation; Randomly select a number of crawled data in the first set and enter them into the comparison set.

5. The method for crawling financial information based on a web crawler according to claim 1, characterized in that: The crawling data collection strategy for the first set includes: Perform a second crawl operation; Obtain the intersection between the first set and the comparison set, recorded as the first intersection set; Eliminate all first intersection set elements in the first set; Randomly select crawled data with a number equal to crawled data threshold a in the first set, and enter them into the first pre-selected set; In the first pre-selected set, each element is selected in turn, and each selected element is compared with all elements in the comparison set by using natural language processing technology; If the comparison result is identical, the currently selected element is marked as a duplicate element; If the comparison result is non-identical, the currently selected element is entered into the comparison set; Get the number of repeated elements; Calculate the ratio between the number of repeated elements and the crawled data threshold a, and record it as the first repetition rate.

6. The method for crawling financial information based on a web crawler according to claim 1, characterized in that: The crawling data collection strategy for the first set includes: Perform the third crawl operation; Obtain the intersection between the first set and the comparison set, recorded as the second intersection set; Eliminate all elements of the second intersection set in the first set; Randomly select crawled data with a number equal to crawled data threshold a in the first set, and enter them into the second pre-selected set; In the second pre-selected set, each element is selected in turn, and each selected element is compared with all elements in the comparison set by using natural language processing technology; If the comparison result is identical, then mark the current element in the second pre-selected set as a duplicate element; If the comparison result is non-identical, the current element in the second pre-selected set is entered into the comparison set; Get the number of repeated elements; Calculate the ratio between the number of repeated elements and the crawled data threshold a, and record it as the second repetition rate; By calculating the first repetition rate and the second repetition rate, the number of crawling times is predicted.

7. A method for crawling financial information based on a web crawler according to claim 6, characterized in that: The method of predicting the number of crawling times by calculating the first repetition rate and the second repetition rate includes: Entering the first repetition rate and the second repetition rate into a repetition set; Calculate the difference between adjacent elements in the repeating set, denoted as b1, b2...bn, where n is the number of elements in the repeating set; The first average repetition rate is calculated by the following formula: In the above formula, d represents the first average repetition rate, bi represents the i-th difference between adjacent elements in the repetition set, i represents the number of elements in the current repetition set, and n represents the number of elements in the repetition set; Set the crawling cutoff threshold e; The estimated number of crawling data processing times is calculated using the following formula: In the above formula, f represents the expected number of crawled data processing times, c represents the number of elements in the repeated set, bn represents the nth element in the repeated set, n is the number of elements in the repeated set, e represents the crawling cutoff threshold, d represents the first average repetition rate, Indicates rounding up.

8. A method for crawling financial information based on a web crawler according to claim 7, characterized in that: The first round of prediction crawling data processing is performed according to the predicted number of crawling times required, including: The first set is crawled f times, and the steps are as follows: S1 obtains the intersection between the first set and the comparison set, and removes all elements of the intersection in the first set; S2 randomly selects crawled data whose number in the first set is the crawled data threshold a, and enters them into the pre-selected set; S3 selects each element in the first pre-selected set in turn by using natural language processing technology, and compares the selected element each time with all elements in the comparison set; S4 If the comparison result is identical, then mark the current element in the pre-selected set as a duplicate element; S5: if the comparison result is non-identical, the current element in the pre-selected set is entered into the comparison set; S6 obtains the number of repeated elements, calculates the ratio between the number of repeated elements and the crawled data threshold a, which is the repetition rate of the current crawled data processing, and enters the repetition rate of the current crawled data processing into the repetition set; S7 records the current crawling processing times, recorded as g; If g is less than f, repeat the above steps S1-S7; If g is equal to f, the crawling operation ends; Sort the elements in the repeating set in the order of entry, obtain the number of elements in the repeating set, record it as m, select the mth element in the repeating set, record it as the comparison element; If the comparison element is less than or equal to the crawling cutoff threshold e, the crawling process is stopped and all elements in the comparison set are retained; If the comparison element is greater than the crawling cutoff threshold e, a second estimated crawling data processing is performed.

9. The method for crawling financial information based on a web crawler according to claim 1, characterized in that: If the second round of prediction crawling data processing is performed, the second round of prediction crawling data processing results are retained, including: The second average repetition rate is calculated by the following formula: In the above formula, r represents the second average repetition rate, bq represents the qth difference between adjacent elements in the repetition set, q represents the number of elements in the current repetition set, and p represents the number of elements in the repetition set; The second estimated crawling data processing times are calculated using the following formula: In the above formula, h represents the second estimated number of crawling data processing times, n represents the number of elements in the repeated set, bp represents the pth element in the repeated set, e represents the crawling cutoff threshold, and r represents the second average repetition rate. Indicates rounding up; Perform h crawling operations on the first set; S1 removes all elements in the comparison set from the first set; S2 randomly selects crawled data whose number in the first set is the crawled data threshold a, and enters them into the pre-selected set; S3 uses natural language processing technology to sequentially judge the elements in the pre-selected set and all the elements in the comparison set; If the judgment result is identical, then the current element in the pre-selected set is marked as a duplicate element; If the judgment result is non-identical, then the current element in the pre-selected set is entered into the comparison set; S6 obtains the number of repeated elements, calculates the ratio between the number of repeated elements and the crawling data threshold a, which is the repetition rate of the current crawling process, and enters the repetition rate of the current crawling process into the repetition set; S7 records the current crawling processing times, recorded as k; If k is less than h, repeat the above steps S1 to S7; If k is equal to h, the crawling process is stopped and all elements in the comparison set are retained.

Images