A dynamic threshold-based acquisition method, device, and medium

Abstract

The application relates to the technical field of electric digital data processing, in particular to a collection method based on a dynamic threshold value, equipment and a medium. The method comprises the following steps: obtaining a set of experimental crawler programs whose crawling results on a target data source are successful, which are included in an experimental collection group; obtaining a crawling success rate of each program in the set on a candidate data source in a candidate data source set; determining a crawling characteristic judgment index of the target data source according to the crawling success rate; if the index is greater than or equal to a preset judgment index threshold value, determining a collection power of the target data source according to a comparison result of a collection power index of the target data source and a preset collection power index threshold value, and collecting the target data source based on the collection power; otherwise, determining the collection power of the target data source according to a comparison result of the collection power index of the target data source and an updated collection power index threshold value. The application can dynamically adjust a decision threshold value, and balance resource efficiency and a collection success rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to a collection method based on dynamic threshold, device and medium. BACKGROUND

[0002] With the explosive growth of Internet information, web crawlers, as a key technology for data acquisition, are widely used in search engines, market analysis, public opinion monitoring and other fields. However, in the actual data collection process, the anti-crawling mechanism of the target data source, network fluctuations, server load and other factors will all cause the success rate of crawling to change dynamically, posing a great challenge to the formulation of collection strategies. Traditional collection methods mostly use static threshold strategies, that is, fixed request frequency, concurrency or timeout time is set for all data sources in advance. Although this method is simple to implement, it cannot adapt to the differences in characteristics of different data sources and real-time changes in network environment, and has limitations: overly aggressive collection strategies are likely to trigger anti-crawling mechanisms, resulting in IP being blocked; while overly conservative strategies will cause resource idling and low collection efficiency. SUMMARY

[0003] The present application aims to provide a collection method based on dynamic threshold, device and medium, to dynamically adjust the decision threshold and achieve a balance between resource efficiency and collection success rate.

[0004] According to a first aspect of the present application, a collection method based on dynamic threshold is provided, the method comprising the following steps:

[0005] Obtaining a set of experimental crawler programs whose crawling results on the target data source are successful, the experimental collection group comprising a plurality of experimental crawler programs.

[0006] Obtaining the crawling success rate of each experimental crawler program in the set of experimental crawler programs on the candidate data source in the set of candidate data sources, the set of candidate data sources comprising a plurality of candidate data sources.

[0007] Determining the crawling characteristic judgment index of the target data source according to the crawling success rate.

[0008] If the crawling characteristic determination index of the target data source is greater than or equal to a preset determination index threshold value, the collection power of the target data source is determined according to a comparison result of the collection power index of the target data source and a preset collection power index threshold value, and the target data source is collected based on the collection power of the target data source; otherwise, the collection power of the target data source is determined according to a comparison result of the collection power index of the target data source and an updated collection power index threshold value, and the target data source is collected based on the collection power of the target data source; the updated collection power index threshold value is obtained according to the preset collection power index threshold value, and the updated collection power index threshold value is not equal to the preset collection power index threshold value.

[0009] Further, the crawling characteristic determination index of the target data source according to the crawling success rate comprises:

[0010] An average value of the crawling success rate of the candidate data source in the candidate data source set by the experimental crawler program in the experimental crawler program set is obtained.

[0011] A variance of the crawling success rate of the candidate data source in the candidate data source set by the experimental crawler program in the experimental crawler program set is obtained.

[0012] The crawling characteristic determination index of the target data source is determined according to the average value and the variance of the crawling success rate; the crawling characteristic determination index of the target data source is negatively correlated with the average value of the crawling success rate; and the crawling characteristic determination index of the target data source is negatively correlated with the variance of the crawling success rate.

[0013] Further, the collection power of the target data source is determined according to a comparison result of the collection power index of the target data source and a preset collection power index threshold value, comprising:

[0014] If the collection power index of the target data source is greater than or equal to a preset first collection power index threshold value, the collection power of the target data source is determined as a preset maximum collection power; if the collection power index of the target data source is less than or equal to a preset second collection power index threshold value, the collection power of the target data source is determined as a preset minimum collection power; and the preset first collection power index threshold value is greater than the preset second collection power index threshold value.

[0015] Further, the collection power of the target data source is determined according to a comparison result of the collection power index of the target data source and an updated collection power index threshold value, comprising:

[0016] If the collection power index of the target data source is greater than or equal to the updated first collection power index threshold, the collection power of the target data source is determined as the preset maximum collection power; if the collection power index of the target data source is less than or equal to the updated second collection power index threshold, the collection power of the target data source is determined as the preset minimum collection power; the updated first collection power index threshold is greater than the updated second collection power index threshold; the updated first collection power index threshold is less than the preset first collection power index threshold, and the updated second collection power index threshold is less than the preset second collection power index threshold.

[0017] Further, the target data source is any candidate data source in a candidate data source set, the candidate data sources in the candidate data source set have the same preset data source attribute, and the preset data source attribute includes at least one of the following attributes: a data source type, an anti-crawling strategy, a page structure type, an access protocol, and a request frequency limit value.

[0018] Further, the preset crawler program attribute of different experimental crawler programs is the same, and the preset crawler program attribute includes at least one of the following attributes: a crawling target, a running parameter, and validity.

[0019] Further, the experimental crawler program and the candidate data source meet a preset condition, and the preset condition includes that the experimental crawler program is compatible with the candidate data source.

[0020] Further, the target data source and the candidate data source are websites.

[0021] According to a second aspect of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned dynamic threshold-based collection method when executing the computer program.

[0022] According to a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the above-mentioned dynamic threshold-based collection method when executed by a processor.

[0023] Compared with the prior art, the present application has at least the following beneficial effects:

[0024] The application obtains the crawling success rate of each experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set, determines the crawling characteristic judgment index of the target data source according to the crawling success rate, if the crawling characteristic judgment index of the target data source is greater than or equal to the preset judgment index threshold value, determines the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold value, and collects the target data source based on the collection power of the target data source; otherwise, determines the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value, and collects the target data source based on the collection power of the target data source. Therefore, by introducing the crawling characteristic judgment index and constructing a dynamic threshold adjustment mechanism, the application can intelligently distinguish the stability characteristics of the target data source, use the conventional threshold for decision-making for the target data source with stable characteristics, use the updated threshold for decision-making for the target data source with unstable characteristics, and the differentiated processing method is beneficial to improving the accuracy of power decision-making and balancing resource efficiency and collection success rate. BRIEF DESCRIPTION OF DRAWINGS DETAILED DESCRIPTION OF THE INVENTION BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0026] Figure 1 The flowchart of the collection method based on the dynamic threshold provided by the first embodiment of the application;

[0027] Figure 2 The flowchart of the step of determining the crawling characteristic judgment index of the target data source provided by the first embodiment of the application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0029] Embodiment one:

[0030] According to the embodiment, as shown in Figure 1 , a collection method based on a dynamic threshold is provided, which comprises the following steps:

[0031] S100, obtaining a set of experimental crawler programs whose crawling results on the target data source are successful from an experimental collection group; the experimental collection group includes a plurality of experimental crawler programs.

[0032] In this embodiment, the experimental collection group is a set composed of a plurality of experimental crawler programs, and the experimental collection group includes different experimental crawler programs. The experimental collection group is used to perform preliminary crawling tests on the candidate data sources in the candidate data source set, and obtain crawling results to evaluate the data source performance. The candidate data source set is a set of a plurality of data sources (such as websites) to be evaluated, and is the target of data collection. The experimental crawler program is a program used to simulate the data collection process, which can initiate a crawling request to the data source and return a success or failure result. The candidate data source is a specific data source to be evaluated, and the target data source is any candidate data source in the candidate data source set, which is the object of evaluation in this embodiment. As a specific implementation, the target data source and the candidate data source are websites.

[0033] In this embodiment, the experimental crawler programs in the experimental collection group and the candidate data sources satisfy a preset condition, and the preset condition includes that the experimental crawler programs are compatible with the candidate data sources. The preset condition that the experimental crawler programs are compatible with the candidate data sources is a basic prerequisite for effective implementation of the experiment. If the crawler and the data source are not compatible, the crawler cannot normally crawl, a large amount of invalid failure data is generated, the set of experimental crawler programs that successfully crawl cannot be screened out, and the entire experimental process is stalled. Based on the preset condition that the experimental crawler programs are compatible with the candidate data sources, it can be ensured that all experimental crawlers can effectively access and crawl the candidate data sources, and invalid samples are avoided.

[0034] In this embodiment, the candidate data sources in the candidate data source set have the same preset data source attribute, and the preset data source attribute includes at least one of the following attributes: data source type, anti-crawling strategy, page structure type, access protocol, and request frequency limit value. The data source type includes static websites, dynamic websites, and the like. The anti-crawling strategy includes verification code verification, IP restriction, User-Agent detection, and the like. The page structure type includes list type structure, detail type structure, nested type structure, and irregular structure. The access protocol includes HTTP, FTP, HTTPS, and the like. The request frequency limit value refers to the maximum number of requests that a data source allows a single client to initiate in a unit of time. In this way, the success rate of the experimental crawler programs can be prevented from being distorted due to differences in the attributes of the data sources, the success rate of the experimental crawler programs can truly reflect the characteristics of the data sources themselves, and an unbiased data sample is provided for subsequent determination of the crawling stability of the target data source based on the success rate of the experimental crawler programs.

[0035] In this embodiment, the preset crawler program attributes of different experimental crawler programs in the experimental collection group are the same, and the preset crawler program attributes include at least one of the following attributes: a crawling target, a running parameter, and validity. The crawling target refers to specific data content that needs to be obtained from a data source, for example, a title, a publication time, and a text of a news website, etc. The running parameter refers to a configuration parameter of the crawler program, including a concurrency number, a timeout time, a retry number, etc. The validity refers to the functional integrity and correctness of the crawler program itself. The preset attributes of the experimental crawler programs are the same, which can exclude the influence of the differences of the crawlers themselves on the experimental results, make the crawling ability and execution logic of all experimental crawler programs consistent, and make the difference of the crawling success rate determined only by the real characteristics of the candidate data source, so that the crawling characteristic judgment index calculated based on the success rate is more objective and truly reflects the stability of the target data source.

[0036] In this embodiment, each experimental crawler program in the experimental collection group is used to perform crawling test on each candidate data source in the candidate data source set, so that the crawling result of each experimental crawler program in the experimental collection group on each candidate data source can be obtained, and the crawling result includes success and failure. A set of experimental crawler programs whose crawling results on the target data source are successful is determined as the experimental crawler program set.

[0037] S200, obtaining a crawling success rate of each experimental crawler program in the experimental crawler program set on a candidate data source in the candidate data source set; the candidate data source set includes a plurality of candidate data sources.

[0038] In this embodiment, the crawling result of each experimental crawler program in the experimental crawler program set on each candidate data source in the candidate data source set is known, and on this basis, the crawling success rate of each experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set can also be obtained. The crawling success rate of any experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set is the ratio of the number of successful crawls of the experimental crawler program on the candidate data source in the candidate data source set to the number of candidate data sources included in the candidate data source set.

[0039] S300, determining a crawling characteristic judgment index of the target data source according to the crawling success rate.

[0040] As a specific implementation, S300 includes, as shown in Figure 2

[0041] S310, obtaining an average value of the crawling success rate of the experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set.

[0042] ​S320, obtaining a variance of the crawling success rate of the experimental crawler program on the candidate data source in the candidate data source set.

[0043] S330, determining a crawling feature judgment index of the target data source according to the mean and the variance of the crawling success rate; the crawling feature judgment index of the target data source is negatively correlated with the mean of the crawling success rate; and the crawling feature judgment index of the target data source is negatively correlated with the variance of the crawling success rate.

[0044] As a specific embodiment, the mean and the variance of the crawling success rate are normalized respectively, and then the difference between 1 and the normalized result of the mean is determined as a first index, and the difference between 1 and the normalized result of the variance is determined as a second index; the product of the first index and the second index is determined as the crawling feature judgment index of the target data source. Those skilled in the art know that the normalization is prior art, which will not be described here.

[0045] In the embodiment, the greater the crawling feature judgment index of the target data source is, the easier the target data source is to be crawled and the more stable the crawling difficulty is; otherwise, the target data source is more difficult to be crawled or the crawling difficulty is less stable.

[0046] S400, if the crawling feature judgment index of the target data source is greater than or equal to a preset judgment index threshold, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and a preset collection power index threshold, and collecting the target data source based on the collection power of the target data source; otherwise, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and an updated collection power index threshold, and collecting the target data source based on the collection power of the target data source; the updated collection power index threshold is obtained according to the preset collection power index threshold, and the updated collection power index threshold is not equal to the preset collection power index threshold.

[0047] As a specific embodiment, the collection power index of the target data source is a comprehensive index quantitatively describing the resource input intensity and the strategy aggressiveness when collecting data of the target data source, and the range of the collection power index of the target data source is [0, 1], which can be manually set.

[0048] As a specific implementation, the determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold value comprises: if the collection power index of the target data source is greater than or equal to a preset first collection power index threshold value, determining the collection power of the target data source as a preset maximum collection power; if the collection power index of the target data source is less than or equal to a preset second collection power index threshold value, determining the collection power of the target data source as a preset minimum collection power; the preset first collection power index threshold value is greater than the preset second collection power index threshold value.

[0049] As a specific implementation, the determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold value further comprises: if the collection power index of the target data source is less than the preset first collection power index threshold value and greater than the preset second collection power index threshold value, performing normalization processing on the collection power index of the target data source according to the preset first collection power index threshold value and the preset second collection power index threshold value, and determining the product of the result of the normalization processing on the collection power index of the target data source and the preset maximum collection power as the collection power of the target data source. It should be understood that the normalization processing here refers to maximum-minimum normalization processing.

[0050] As a specific implementation, the determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value comprises: if the collection power index of the target data source is greater than or equal to an updated first collection power index threshold value, determining the collection power of the target data source as a preset maximum collection power; if the collection power index of the target data source is less than or equal to an updated second collection power index threshold value, determining the collection power of the target data source as a preset minimum collection power; the updated first collection power index threshold value is greater than the updated second collection power index threshold value; the updated first collection power index threshold value is less than the preset first collection power index threshold value, and the updated second collection power index threshold value is less than the preset second collection power index threshold value.

[0051] As a specific implementation, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold further comprises: if the collection power index of the target data source is less than the updated first collection power index threshold and greater than the updated second collection power index threshold, normalizing the collection power index of the target data source according to the updated first collection power index threshold and the updated second collection power index threshold, and determining the larger value of the product of the normalized result of the collection power index of the target data source and the preset maximum collection power and the original collection power as the collection power of the target data source; wherein the original collection power is: the product of the normalized result of the collection power index of the target data source according to the preset first collection power index threshold and the preset second collection power index threshold and the preset maximum collection power. It should be understood that the normalization here refers to maximum-minimum normalization.

[0052] As a specific implementation, T'1 is the updated first collection power index threshold, 0 < T'1 < 1, T'1 = (1 - k1 × (k - k0) / k) × T1, wherein T1 is the preset first collection power index threshold, 0 < T1 < 1, k1 is the preset first adjustment coefficient, 0 < k1 < 1, k is the preset determination index threshold, k0 is the crawling feature determination index of the target data source, k0 < k. Optionally, T1, k1, and k are empirical values, for example, T1 = 0.8, k1 = 0.5, and k = 0.6. Based on the formula, when the actual crawling feature determination index of the target data source is less than the preset determination index threshold, the updated first collection power index threshold is less than the preset first collection power index threshold; and the more the actual crawling feature determination index of the target data source is less than the preset determination index threshold, the more the updated first collection power index threshold is less than the preset first collection power index threshold, which helps to enable the strong mode earlier to cope with possible crawling difficulties when the data source is unstable or difficult to be crawled.

[0053] As a specific embodiment, T'2 is an updated second acquisition power index threshold, 0 < T'2 < 1, T'2 = (1 - k2 x (k - k0) / k) x T2, wherein T2 is a preset second acquisition power index threshold, 0 < T2 < 1, k2 is a preset second adjustment coefficient, 0 < k2 < 1, k is a preset judgment index threshold, k0 is a crawling characteristic judgment index of the target data source, k0 < k, and (1 - k1) x T1 > (1 - k2) x T2. Optionally, T2 and k2 are empirical values, for example, T2 = 0.3 and k2 = 0.3. Based on the formula, the minimum power acquisition instruction is triggered only when the acquisition power index decreases to a lower level, which forces the system to adopt a strategy of increasing the sampling power when the data source is unstable or difficult to be crawled, and the system is less likely to enter a low-power or sleep state, thereby giving priority to guaranteeing the crawling success.

[0054] The embodiment obtains the crawling success rate of each experimental crawler program in the experimental crawler program set to each candidate data source in the candidate data source set, determines the crawling characteristic judgment index of the target data source according to the crawling success rate, and if the crawling characteristic judgment index of the target data source is greater than or equal to the preset judgment index threshold, determines the acquisition power of the target data source according to the comparison result of the acquisition power index of the target data source and the preset acquisition power index threshold, and collects the target data source based on the acquisition power of the target data source; otherwise, determines the acquisition power of the target data source according to the comparison result of the acquisition power index of the target data source and the updated acquisition power index threshold, and collects the target data source based on the acquisition power of the target data source. Thus, the embodiment can intelligently distinguish the stability characteristics of the data source by introducing the crawling characteristic judgment index and constructing a dynamic threshold adjustment mechanism, and can make decisions using a conventional threshold for data sources with stable characteristics and using an updated threshold for data sources with unstable characteristics. This differentiated processing method is conducive to improving the accuracy of power decision-making, thereby improving the success rate and efficiency of data collection as a whole.

[0055] Embodiment two:

[0056] Compared with embodiment one, the embodiment further provides a process for determining the acquisition power index of the target data source, which includes:

[0057] S410, periodically probe the target data source by the experimental collection group according to a preset probing period, and obtain a probing result corresponding to each time of probing according to the crawling result of each experimental crawler program to the target data source corresponding to each time of probing; the probing result corresponding to each time of probing includes the number of successful crawls and the crawling success rate; the crawling result of any experimental crawler program to the target data source corresponding to each time of probing is success or failure.

[0058] In this embodiment, the detection period refers to the time interval between two adjacent detections, and the detection period is used to control the detection frequency of the target data source. The preset detection period is a pre-set time parameter. Optionally, the preset detection period is an empirical value, for example, 10 minutes / time or 30 minutes / time. In each detection period, the experimental crawler program is controlled to initiate a standardized crawling request to the target data source, and the crawling result (success / failure) is recorded.

[0059] In this embodiment, the periodic detection refers to a process of periodically initiating a standardized crawling request to the target data source by the experimental collection group according to the preset detection period. For each detection, the number of experimental crawler programs in the experimental collection group corresponding to the detection and having a successful crawling result to the target data source is determined as the crawling success number of the target data source. The ratio of the crawling success number of the target data source to the number of experimental crawler programs included in the experimental collection group is determined as the crawling success rate of the target data source.

[0060] In this embodiment, the crawling success number and the success rate of the target data source are used to evaluate the carrying capacity and the crawlable performance of the target data source, and the carrying capacity and the crawlable performance of the target data source are converted into quantifiable indexes, thereby providing data basis for subsequent determination of the collection power index of the target data source.

[0061] S420, obtaining the latest crawling success number and the latest crawling success rate of the target data source according to the detection result corresponding to the latest preset detection.

[0062] In this embodiment, the preset number of times refers to the number of the latest detection results referred to when the latest crawling success number and the latest crawling success rate are calculated, which is a pre-set number parameter. The preset number of times is used to filter historical obsolete data, focus on the recent state of the data source, and balance timeliness and stability. Optionally, the preset number of times is an empirical value, for example, the preset number of times is 1 or 3 or 5.

[0063] As a specific implementation, the latest crawling success number is B, B=∑ N j=1 (δ j ×B j ), B j is the crawling success number corresponding to the latest jth detection, δ j is the weight corresponding to the latest jth detection, j has a value range of 1 to N, and N is the preset number of times; optionally, δ j =1 / N. The latest crawling success rate is p, p=∑ N j=1 (δ j ×p j ), p j is the crawling success rate corresponding to the latest jth detection.

[0064] S430, acquiring the collection power index of the target data source according to the latest successful crawling number, the latest successful crawling rate and the priority of the target data source.

[0065] In this embodiment, the collection power index of the target data source is an index for comprehensively measuring the priority, the latest successful crawling number and the latest successful crawling rate of the target data source. For the target data source, the successful crawling number of the target data source is positively correlated with the collection power index of the target data source, the successful crawling rate of the target data source is negatively correlated with the collection power index of the target data source, and the priority of the target data source is positively correlated with the collection power index of the target data source.

[0066] As a specific implementation, S430 includes:

[0067] S431, determining the ratio of the first dynamic coefficient and the latest successful crawling rate of the target data source as a first collection power sub-index of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0068] In this embodiment, the first dynamic coefficient is a dynamic adjustment parameter related to the latest successful crawling rate of the target data source. As a specific implementation, the first dynamic coefficient is positively correlated with the latest successful crawling rate of the target data source.

[0069] In this embodiment, the first collection power sub-index reflects the contribution of the successful rate to the collection power index, and the first collection power sub-index is I1, I1=α / p, and the value range of I1 is [0, 1].

[0070] As a specific implementation, the first dynamic coefficient is α, α=(e b×(p-1) -e -b ) / (1-e -b ), p is the latest successful crawling rate of the target data source, and b is a preset parameter value, b>0. Optionally, b is an empirical value, for example, b=2. Based on the formula, α increases with the increase of the latest successful rate p of the target data source, when p→0 (the latest successful rate of the target data source is extremely low), α≈0, so I1≈0, which can significantly reduce the weight of the target data source, which is consistent with the scene that unstable data source should reduce resources; when p is at a medium level, α grows with p but the growth rate is slower than p, so I1=α / p grows with p but the growth rate is gentle, avoiding that the target data source obtains too high weight at a medium successful rate; when p→1 (the successful rate is extremely high), α≈1, so I1≈1, which can allocate a larger weight to the target data source, and does not additionally amplify the weight. Based on the formula, non-linear adjustment is realized through an exponential function, the punishment for low successful rate data source is more significant (α is much smaller than p), the reward for high successful rate data source is more restrained (α is close to p), avoiding that resources are tilted to extremely unstable or excessively compatible data sources.

[0071] S432, determining a second collection power sub-index of the target data source as a ratio of the latest successful crawling number of the target data source and the second dynamic coefficient; the second collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0072] In the embodiment, the second dynamic coefficient is a dynamic adjustment parameter related to the maximum value of the historical successful number of the target data source. As a specific embodiment, for the target data source, the second dynamic coefficient is determined according to the maximum value of the successful crawling number of the target data source in the target historical time period.

[0073] In the embodiment, the second collection power sub-index reflects the contribution of the successful number to the collection power index, and the second collection power sub-index is I2, I2=B / β, B is the latest successful crawling number of the target data source, and the value range of the second collection power sub-index is [0, 1].

[0074] As a specific embodiment, β=w1×B 1,max +(1-w1)×B 2,max . B 1,max is the maximum value of the successful crawling number of the target data source in a specified unit time period, B 2,max is the maximum value of the successful crawling number of the target data source in other unit time periods except the specified unit time period in the target historical time period, the specified unit time period is the unit time period closest to the current time; w1 is a preset weight, and optionally, 0.5<w1<1. Based on the formula, β integrates the best performance of the recent (B1,max) and the history (B2,max), which can avoid misjudgment caused by single time period data (such as short-term fluctuations or obsolete data), and w1>0.5 ensures that the recent performance weight is higher; β updates with the historical performance of the data source, so that I2 can reflect the position of the current value of the data source relative to the historical best level, rather than the absolute value under the fixed benchmark; the essence of I2=B / β is the ratio of the current performance to the historical best performance, when B is close to the recent or historical maximum value, I2→1 (indicating excellent current performance); when B is much smaller than the historical maximum value, I2→0 (indicating poor current performance), the embodiment measures the value of the current successful number B through β, which not only pays attention to the recent performance of the data source, but also considers its long-term stability, and finally makes I2 reflect the advantages and disadvantages of the current successful number relative to the historical best level.

[0075] As a specific embodiment, the target historical time period includes multiple unit times, and the target data source is periodically detected multiple times in any unit time, optionally, the target historical time period is 7 days, the unit time period is 1 day, the target data source is detected every 30 minutes (i.e. the detection period is 30 minutes), and each detection can obtain a successful crawling number.

[0076] S433, acquire the acquisition power index of the target data source according to the first acquisition power sub-index, the second acquisition power sub-index and the priority of the target data source; the acquisition power index of the target data source is positively correlated with the first acquisition power sub-index, and the acquisition power index of the target data source is positively correlated with the second acquisition power sub-index.

[0077] In this embodiment, the priority reflects the importance of the data source, and the higher the importance, the greater the priority; the value range of the priority is [0, 1]. Optionally, the priority of the target data source is manually set.

[0078] As a specific implementation, the product of the first acquisition power sub-index, the second acquisition power sub-index and the priority of the target data source is determined as the acquisition power index of the target data source.

[0079] In this embodiment, the first acquisition power sub-index, the second acquisition power sub-index and the priority are cooperatively evaluated in three dimensions, and finally an index (i.e., the acquisition power index) that comprehensively reflects the degree of worthiness of the data source to be collected is output, which avoids the single dimension dominating the resource allocation and improves the comprehensiveness of the evaluation, so that the allocation of the data collection resource is more accurate, more dynamic and more in line with the actual demand.

[0080] Based on this embodiment, the target data source is periodically detected through a preset detection period, and the latest crawling success number and success rate are calculated based on the latest preset detection result, so that the state change (such as anti-crawling strategy upgrade, server bearing capacity fluctuation, temporary failure, etc.) of the data source can be captured in real time. Compared with the lag response of the traditional fixed crawler strategy to the state change, this embodiment can keep the acquisition strategy synchronized with the current state of the target data source, and improve the adaptability of data collection to dynamic environment. Based on this embodiment, through the whole-process automation mechanism of periodic detection and automatic calculation of the latest state index, the dependence on manual intervention is reduced, which is especially suitable for multi-data source and high-dynamic collection scenarios, and the labor cost is significantly reduced.

[0081] Embodiment three:

[0082] Compared with embodiment two, this embodiment further provides a process of determining the priority of the target data source, and the process comprises:

[0083] G100, acquire the data access heat of the target data source according to the number of times that the data of the target data source is accessed within a target time period; the target time period is a time period with a preset length ending at the current time.

[0084] In this embodiment, the preset length is an empirical value, which can be flexibly set according to the business scenario (such as 1 day, 7 days, etc.), for example, for a scenario with high real-time requirement, it can be set to 1 day, and for a scenario with low real-time requirement, it can be set to 7 days.

[0085] In this embodiment, the target data source refers to an object that needs to be collected, and the target data source can generate data. Optionally, the target data source is a website.

[0086] In this embodiment, the data access heat of the target data source refers to the frequency of access to the target data source, and the data access heat of the target data source is positively correlated with the number of times of access to the data of the target data source in the target time period. As a specific implementation, the number of times of access to the data of the target data source in the target time period is normalized to obtain the data access heat of the target data source. Optionally, the data access heat of the target data source is H, H = (c - c min ) / (c max -c min ), where c is the number of times of access to the data of the target data source in the target time period, c min and c max are the preset minimum access times and maximum access times, respectively. Thus, the number of times of access to the data of the target data source in the target time period can be standardized to the interval [0, 1].

[0087] G200, obtaining the data update activity of the target data source according to the update frequency of the target data source in the target time period.

[0088] In this embodiment, the data update activity of the target data source reflects the dynamic characteristics (average level and change trend) of the data update of the target data source, and the dynamic characteristics of the data update of the target data source directly affect the data timeliness. As a specific implementation, G200 includes:

[0089] G210, obtaining the update frequency curve of the target data source in the target time period according to the update frequency of the target data source in the target time period; the update frequency of the target data source in the target time period includes the update frequency of the target data source corresponding to each unit time in the target time period; the horizontal axis of the update frequency curve of the target data source in the target time period is time, and the vertical axis of the update frequency curve of the target data source in the target time period is update frequency.

[0090] As a specific implementation, the update frequency (i.e., the number of data updates) of the target data source in each unit time included in the target time period is counted, and the update frequency curve is drawn with time as the horizontal axis and update frequency as the vertical axis. Based on the update frequency curve, the change trend of the update frequency of the target data source over time can be obtained.

[0091] G220, obtaining the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period.

[0092] In this embodiment, the absolute value of the slope of the update frequency curve of the target data source in the target time period can reflect the fluctuation degree of the update frequency, and the average value of the absolute value of the slope can quantify the overall fluctuation trend. The greater the average value of the absolute value of the slope, the more intense the update frequency fluctuation, and the stronger the information capturing ability of the target data source.

[0093] G230, obtaining the data update activity of the target data source according to the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period and the average value of the update frequency of the target data source in different unit time in the target time period.

[0094] In this embodiment, the average level of the update frequency (basic activity) and the fluctuation trend (dynamic activity) are combined to comprehensively evaluate the update characteristics of the target data source. As a specific implementation, G230 includes:

[0095] G231, normalizing the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period to obtain a first activity of the target data source.

[0096] As a specific implementation, the first activity is A1, A1=(z-z min ) / (z max -z min ), where z is the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period, z min and z max are the minimum and maximum values of the average value of the absolute value of the slope, respectively. In this way, the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period can be standardized to the interval [0, 1].

[0097] G232, normalizing the average value of the update frequency of the target data source in different unit time in the target time period to obtain a second activity of the target data source.

[0098] As a specific implementation, the second activity is A2, A2=(f-f min ) / (f max -f min ), where f is the average value of the update frequency of the target data source in different unit time in the target time period, f min and f max are the minimum and maximum values of the average value of the update frequency, respectively. In this way, the average value of the update frequency of the target data source in different unit time in the target time period can be standardized to the interval [0, 1].

[0099] G233, obtaining the data update activity of the target data source according to the weight corresponding to the slope of the preset update frequency curve, the first activity of the target data source, the weight corresponding to the average value of the preset update frequency, and the second activity of the target data source.

[0100] As a specific embodiment, the data update activity of the target data source is A, A = w k ×A1+w f ×A2, w k is the weight corresponding to the slope of the preset update frequency curve, w f is the weight corresponding to the average value of the preset update frequency, w k +w f =1, and w k >0, w f >0. w k reflects the importance of the update frequency fluctuation trend, w f reflects the importance of the average update level, which can be set according to experience, for example, w k =w f =0.5, or w k =0.4, w f =0.6.

[0101] Thus, the embodiment comprehensively evaluates the data update activity of the target data source by combining dynamic and static update characteristics.

[0102] G300, obtaining the output data amount index of the target data source according to the amount of data generated by the target data source in a target time period.

[0103] In the embodiment, the output data amount index of the target data source reflects the amount of data generated by the target data source in the target time period, and the output data amount index of the target data source is positively correlated with the amount of data generated by the target data source in the target time period. The greater the output data amount of the target data source, the richer the information contained in the target data source.

[0104] As a specific embodiment, the total data amount v generated by the target data source in the target time period is counted (for example, in bytes), and the output data amount index v idx of the target data source is obtained by normalization processing, v idx =(v-v min ) / (v max -v min ), where v min and v max are respectively preset minimum data amount and maximum data amount. Thus, the amount of data generated by the target data source in the target time period can be standardized to the interval [0, 1].

[0105] G400, obtaining a data collection resource consumption index of the target data source according to resources consumed in collecting data of the target data source in the target time period.

[0106] In this embodiment, the data collection resource consumption index of the target data source reflects how many resources are consumed in collecting data of the target data source, and the data collection resource consumption index of the target data source is positively correlated with resources consumed in collecting data of the target data source in the target time period. The more resources consumed in the collection process, the higher the collection cost, which needs to be balanced in the priority evaluation.

[0107] As a specific implementation, each preset type of resource consumed in collecting the target data source in the target time period is counted, each preset type of resource is normalized, and the normalized results are weighted and summed to obtain the resource consumption index. Optionally, the preset type of resource includes CPU occupancy, memory occupancy, and bandwidth occupancy, normalization uses an existing normalization method, for example, a min-max normalization method; the weight of each preset type of resource corresponding to the weighted sum of the normalized results is an empirical value, the weight of each preset type of resource is greater than 0, the sum of the weights of each preset type of resource is 1, for example, the weight of each preset type of resource is 1 / R, and R is the number of preset types of resources.

[0108] G500, obtaining the priority of the target data source according to the data access heat, the data update activity, the output data volume index, and the data collection resource consumption index of the target data source.

[0109] In this embodiment, the priority of the target data source comprehensively evaluates the importance of the target data source, and as a specific implementation, G500 includes:

[0110] G510, obtaining a weight corresponding to the data access heat, a weight corresponding to the data update activity, a weight corresponding to the output data volume, and a weight corresponding to the data collection resource consumption; the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption are all greater than 0.

[0111] As a specific implementation, obtaining the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption includes:

[0112] G511, obtaining a target variable; the target variable is a manually labeled priority.

[0113] As a specific implementation, the manually labeled priority is collected as the target variable Y (labeled by a domain expert according to business requirements on historical data sources, with a value range of [0, 1]).

[0114] G512, taking four indicators of the target data source as characteristic variables, constructing a data set containing the corresponding relationship between the characteristic variables and the target variable; the four indicators include data access heat, data update activity, output data volume index, and data collection resource efficiency.

[0115] As a specific embodiment, a data set is constructed, each piece of data including a characteristic variable and a corresponding target variable. The data collection resource efficiency is the difference between 1 and the data collection resource consumption index.

[0116] G513, training a linear regression model based on the data set to obtain the coefficients corresponding to each characteristic variable in the linear regression model.

[0117] As a specific embodiment, a linear regression model is trained, and the model expression is Y = w h x1 + w a x2 + w v x3 + w r x4, w h is the characteristic coefficient corresponding to the data access heat, w a is the characteristic coefficient corresponding to the data update activity, w v is the characteristic coefficient corresponding to the output data volume index, and w r is the characteristic coefficient corresponding to the data collection resource consumption index. x1, x2, x3, and x4 are respectively the characteristic variable corresponding to the data access heat, the characteristic variable corresponding to the data update activity, the characteristic variable corresponding to the output data volume index, and the characteristic variable corresponding to the data collection resource efficiency. During the training process, non-negative constraints are added to w h , w a , w v , and w r to ensure that w h , w a , w v , and w r are positive; the linear regression model is trained by the least square method. Those skilled in the art know that the specific training process is prior art, which will not be described here.

[0118] G514, normalizing the coefficients to make the sum of the normalized coefficients equal to 1 and each coefficient greater than 0, and determining the normalized coefficients as the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption, respectively.

[0119] As a specific embodiment, the weight corresponding to the data access heat is w h , w h = w h / (wh +w a +w v +w r ); the weight corresponding to the data update activity is w a , w a = w a / (w h +w a +w v +w r ); the weight corresponding to the output data volume is w v , w v = w v / (w h +w a +w v +w r ); the weight corresponding to the data collection resource consumption is w r , w r = w r / (w h +w a +w v +w r ).

[0120] Based on G511-G514, the embodiment determines the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume and the weight corresponding to the data collection resource consumption in a data-driven manner, and the weights thus determined can be more in line with actual business scenarios.

[0121] G520, the product of the weight corresponding to the data access heat and the data access heat of the target data source is determined as the first sub-priority of the target data source.

[0122] G530, the product of the weight corresponding to the data update activity and the data update activity of the target data source is determined as the second sub-priority of the target data source.

[0123] G540, the product of the weight corresponding to the output data volume and the output data volume index of the target data source is determined as the third sub-priority of the target data source.

[0124] G550, the product of the weight corresponding to the data collection resource consumption and the data collection resource efficiency of the target data source is determined as the fourth sub-priority of the target data source; the data collection resource efficiency of the target data source is the difference between 1 and the data collection resource consumption index of the target data source.

[0125] G560, the sum of the first sub-priority of the target data source, the second sub-priority of the target data source, the third sub-priority of the target data source and the fourth sub-priority of the target data source is determined as the priority of the target data source.

[0126] Based on G520-G560, the priority of the target data source is obtained by weightedly fusing the data access heat, data update activity, output data volume index and data collection resource efficiency of the target data source, so that the priority of the target data source determined considers more comprehensive dimensions.

[0127] Embodiment four:

[0128] The embodiment provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0129] Obtain a set of experimental crawler programs whose crawling results on the target data source are successful, included in an experimental collection group; the experimental collection group includes a plurality of experimental crawler programs.

[0130] Obtain a crawling success rate of each experimental crawler program in the set of experimental crawler programs on a candidate data source in a set of candidate data sources; the set of candidate data sources includes a plurality of candidate data sources.

[0131] Determine a crawling characteristic judgment index of the target data source according to the crawling success rate.

[0132] If the crawling characteristic judgment index of the target data source is greater than or equal to a preset judgment index threshold value, determine the collection power of the target data source according to a comparison result of the collection power index of the target data source and the preset collection power index threshold value, and collect the target data source based on the collection power of the target data source; otherwise, determine the collection power of the target data source according to a comparison result of the collection power index of the target data source and an updated collection power index threshold value, and collect the target data source based on the collection power of the target data source; the updated collection power index threshold value is obtained according to the preset collection power index threshold value, and the updated collection power index threshold value is not equal to the preset collection power index threshold value.

[0133] Embodiment five:

[0134] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0135] Obtain a set of experimental crawler programs whose crawling results on the target data source are successful, included in an experimental collection group; the experimental collection group includes a plurality of experimental crawler programs.

[0136] Obtain a crawling success rate of each experimental crawler program in the set of experimental crawler programs on a candidate data source in a set of candidate data sources; the set of candidate data sources includes a plurality of candidate data sources.

[0137] The crawling characteristic judgment index of the target data source is determined according to the crawling success rate.

[0138] If the crawling characteristic judgment index of the target data source is greater than or equal to a preset judgment index threshold value, the collection power of the target data source is determined according to a comparison result of the collection power index of the target data source and a preset collection power index threshold value, and the target data source is collected based on the collection power of the target data source; otherwise, the collection power of the target data source is determined according to a comparison result of the collection power index of the target data source and an updated collection power index threshold value, and the target data source is collected based on the collection power of the target data source; the updated collection power index threshold value is obtained according to the preset collection power index threshold value, and the updated collection power index threshold value is not equal to the preset collection power index threshold value.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0140] Although some specific embodiments of the present application have been described in detail by examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A dynamic threshold based acquisition method, characterized in that, The method comprises the following steps: obtaining a set of experimental crawler programs whose crawling results on a target data source are successful, wherein the experimental set comprises a plurality of experimental crawler programs; obtaining a crawling success rate of each experimental crawler program in the set of experimental crawler programs on a candidate data source in a set of candidate data sources, wherein the set of candidate data sources comprises a plurality of candidate data sources; determining a crawling characteristic judgment index of the target data source according to the crawling success rate, wherein the crawling characteristic judgment index of the target data source is negatively correlated with an average value of the crawling success rate and is negatively correlated with a variance of the crawling success rate; if the crawling characteristic judgment index of the target data source is greater than or equal to a preset judgment index threshold value, determining a collection power of the target data source according to a comparison result of a collection power index of the target data source and a preset collection power index threshold value, and collecting the target data source based on the collection power of the target data source; otherwise, determining the collection power of the target data source according to a comparison result of the collection power index of the target data source and an updated collection power index threshold value, and collecting the target data source based on the collection power of the target data source, wherein the updated collection power index threshold value is obtained according to the preset collection power index threshold value and is not equal to the preset collection power index threshold value, the collection power index of the target data source is an index for comprehensively measuring a priority of the target data source, a latest crawling success number and a latest crawling success rate, the latest crawling success number of the target data source is positively correlated with the collection power index of the target data source, the latest crawling success rate of the target data source is negatively correlated with the collection power index of the target data source, and the priority of the target data source is positively correlated with the collection power index of the target data source.

2. The dynamic threshold based acquisition method of claim 1, wherein, The method comprises the following steps: obtaining an average value of the crawling success rate of the experimental crawler programs in the set of experimental crawler programs on the candidate data sources in the set of candidate data sources; obtaining a variance of the crawling success rate of the experimental crawler programs in the set of experimental crawler programs on the candidate data sources in the set of candidate data sources; determining the crawling characteristic judgment index of the target data source according to the average value and the variance of the crawling success rate.

3. The dynamic threshold based acquisition method of claim 1, wherein, The method comprises the following steps: if the collection power index of the target data source is greater than or equal to a preset first collection power index threshold value, determining the collection power of the target data source as a preset maximum collection power; if the collection power index of the target data source is less than or equal to a preset second collection power index threshold value, determining the collection power of the target data source as a preset minimum collection power, wherein the preset first collection power index threshold value is greater than the preset second collection power index threshold value.

4. The dynamic threshold based acquisition method of claim 3, wherein, The method comprises the following steps: If the collection power index of the target data source is greater than or equal to the updated first collection power index threshold, the collection power of the target data source is determined as the preset maximum collection power; if the collection power index of the target data source is less than or equal to the updated second collection power index threshold, the collection power of the target data source is determined as the preset minimum collection power; the updated first collection power index threshold is greater than the updated second collection power index threshold; the updated first collection power index threshold is less than the preset first collection power index threshold, and the updated second collection power index threshold is less than the preset second collection power index threshold.

5. The dynamic threshold based acquisition method of claim 1, wherein, The target data source is any candidate data source in a candidate data source set, the candidate data sources in the candidate data source set have the same preset data source attribute, and the preset data source attribute includes at least one of the following attributes: data source type, anti-crawling strategy, page structure type, access protocol, and request frequency limit value.

6. The dynamic threshold based acquisition method of claim 1, wherein, The preset crawler program attributes of different experimental crawler programs are the same, and the preset crawler program attribute includes at least one of the following attributes: crawling target, running parameter, and validity.

7. The dynamic threshold based acquisition method of claim 1, wherein, The experimental crawler program and the candidate data source meet a preset condition, and the preset condition includes that the experimental crawler program is compatible with the candidate data source.

8. The dynamic threshold based acquisition method of claim 1, wherein, The target data source and the candidate data source are websites.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the dynamic threshold-based collection method in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the dynamic threshold-based collection method in any one of claims 1 to 8.

Images