A data acquisition processing method, device and medium

Abstract

The application relates to the technical field of electric digital data processing, in particular to a data acquisition processing method and device and a medium. The method comprises the following steps: acquiring an experimental acquisition group; periodically detecting a target data source through the experimental acquisition group according to a preset detection period, and acquiring a detection result corresponding to each detection according to a crawling result of each experimental crawler program corresponding to the detection; acquiring a latest crawling success number and a latest crawling success rate of the target data source according to a detection result corresponding to the latest preset detection; acquiring an acquisition power index of the target data source according to the latest crawling success number, the latest crawling success rate and a priority of the target data source; determining an acquisition power of the target data source according to the acquisition power index of the target data source, and performing data acquisition on the target data source based on the acquisition power of the target data source. The application realizes dynamic matching of the acquisition power of the data source and the state of the data source.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to a data collection processing method, device and medium. BACKGROUND

[0002] With the deep application of big data technology in various fields, data has become the core asset of enterprise decision-making and business optimization, and efficient data collection is the premise of realizing the value of big data. Web crawler program, as the core tool of automated data collection, is widely used to obtain target data from various network data sources (such as websites, etc.). For example, the Chinese patent application with the application number 202510450850.X discloses a self-adaptive data collection method and device, which can automatically adjust the basic crawler program according to the different web pages to generate a customized version of the crawler of the web page, thereby improving the accuracy and efficiency of crawling.

[0003] Although the above patent application can match the crawler and formulate the corresponding crawling strategy for different web pages (i.e. data sources), it does not consider the case that the state of the web page itself is also likely to change, resulting in that the previously formulated crawling strategy for the web page is no longer applicable to the current web page. It can be seen that the traditional data collection usually cannot perceive the state change of the data source (such as concurrent request carrying capacity, anti-crawler strategy upgrade, etc.) in real time, resulting in that the resources are still blindly allocated when the data source is unavailable or responds slowly, causing waste of bandwidth and computing resources. How to realize the dynamic matching of the collection frequency of the data source and the state of the data source is a problem to be solved. SUMMARY

[0004] The present application aims to provide a data collection processing method, device and medium to realize the dynamic matching of the collection frequency of the data source and the state of the data source.

[0005] According to a first aspect of the present application, a data collection processing method is provided, which comprises the following steps:

[0006] An experimental collection group is obtained; the experimental collection group comprises a plurality of experimental crawler programs.

[0007] According to a preset detection period, the target data source is periodically detected by the experimental collection group, and the detection result corresponding to each detection is obtained according to the crawling result of the target data source by any experimental crawler program; the detection result corresponding to any detection includes the number of successful crawling and the success rate of crawling; the crawling result of the target data source by any experimental crawler program corresponding to any detection is success or failure.

[0008] The latest number of successful crawling and the latest success rate of crawling of the target data source are obtained according to the detection result corresponding to the latest preset detection.

[0009] According to the latest crawling success number, the latest crawling success rate and the priority of the target data source, the collection power index of the target data source is obtained.

[0010] According to the collection power index of the target data source, the collection power of the target data source is determined, and data collection is performed on the target data source based on the collection power of the target data source.

[0011] Further, the crawling result of each experimental crawler program corresponding to the target data source in the detection is obtained, and the detection result corresponding to the detection is obtained.

[0012] For any detection, the number of experimental crawler programs whose crawling result on the target data source is successful in the experimental collection group corresponding to the detection is determined as the crawling success number of the target data source; and the ratio of the crawling success number of the target data source to the number of experimental crawler programs included in the experimental collection group is determined as the crawling success rate of the target data source.

[0013] Further, the crawling success number of the target data source is positively correlated with the collection power index of the target data source, the crawling success rate of the target data source is negatively correlated with the collection power index of the target data source, and the priority of the target data source is positively correlated with the collection power index of the target data source.

[0014] Further, according to the latest crawling success number, the latest crawling success rate and the priority of the target data source, the collection power index of the target data source is obtained, including:

[0015] The ratio of the first dynamic coefficient to the latest crawling success rate of the target data source is determined as the first collection power sub-index of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0016] The ratio of the latest crawling success number of the target data source to the second dynamic coefficient is determined as the second collection power sub-index of the target data source; the second collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0017] According to the first collection power sub-index, the second collection power sub-index and the priority of the target data source, the collection power index of the target data source is obtained; the collection power index of the target data source is positively correlated with the first collection power sub-index, and the collection power index of the target data source is positively correlated with the second collection power sub-index.

[0018] Further, the first dynamic coefficient is positively correlated with the latest crawling success rate of the target data source.

[0019] Further, the second dynamic coefficient is determined according to the maximum value of the crawling success number of the target data source in the target historical time period.

[0020] Further, the product of the first acquisition power sub-index, the second acquisition power sub-index and the priority of the target data source is determined as an acquisition power index of the target data source.

[0021] Further, the acquisition power index of the target data source is positively correlated with the acquisition power of the target data source.

[0022] According to a second aspect of the present application, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned data acquisition processing method when executing the computer program.

[0023] According to a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the above-mentioned data acquisition processing method when executed by a processor.

[0024] Compared with the prior art, the present application has at least the following beneficial effects:

[0025] The present application periodically detects the target data source according to the preset detection period, and calculates the latest crawling success number and success rate based on the latest preset detection result, so as to capture the state change (such as anti-crawling strategy upgrade, server carrying capacity fluctuation, temporary failure, etc.) of the data source in real time. Compared with the lag response of the traditional fixed crawler strategy to the state change, the present application can keep the acquisition strategy synchronized with the current state of the target data source, and improve the adaptability of data acquisition to the dynamic environment.

[0026] The present application calculates the acquisition power index based on the latest crawling success number (reflecting the current carrying upper limit of the data source) and the latest crawling success rate (reflecting the current anti-crawling strength), and can accurately match the real-time accessibility of the data source: for the data source in good state (high success rate and strong carrying capacity), higher acquisition power can be allocated to fully utilize its resources; for the data source in poor state (low success rate and frequent interception), the acquisition power is automatically reduced to reduce invalid requests (such as intercepted crawler resources and wasted bandwidth). This mechanism solves the problems of resource waste in poor state and insufficient acquisition in good state in the traditional fixed strategy, and improves the overall resource utilization efficiency.

[0027] The application integrates the target data source priority and the latest state index (i.e. the latest crawling success number and the latest crawling success rate) into the calculation of the collection power index, so as to ensure that the high-priority data source obtains reasonable resource inclination within the range allowed by its current state: when the high-priority data source is in good condition, resources are preferentially allocated to ensure efficient collection of core data; when the high-priority data source is in poor condition, invalid consumption caused by blind investment of resources is avoided, and dynamic adjustment based on its actual accessibility is performed, so as to balance the priority weight and real-time feasibility and improve the stability and reliability of core data collection.

[0028] Therefore, the application reduces the dependence on manual intervention through the full-process automation mechanism of periodic detection, automatic calculation of the latest state index and dynamic generation of the collection power, is especially suitable for multi-data-source and high-dynamic collection scenarios, and significantly reduces the labor cost. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 A flowchart of the data collection processing method provided for the first embodiment of the application;

[0031] Figure 2 A flowchart of the step of obtaining the collection power index of the target data source provided for the first embodiment of the application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the application will be described clearly and completely with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0033] Embodiment one:

[0034] According to the present embodiment, as shown in Figure 1 , a data collection processing method is provided, which comprises the following steps:

[0035] S100, obtaining an experimental collection group; the experimental collection group comprises a plurality of experimental crawlers.

[0036] In this embodiment, the experimental collection group is a set composed of a plurality of experimental crawlers, and the experimental collection group includes different experimental crawlers. The experimental collection group is used to perform preliminary crawling tests on the target data source, and obtain crawling results to evaluate the performance of the target data source. The experimental crawler is a program used to simulate the data collection process, which can initiate a crawling request to the target data source and return a success or failure result.

[0037] The target data source is a data source to be evaluated (such as a website), which is the target of data collection.

[0038] In this embodiment, the crawling test on the target data source based on different experimental crawlers can improve the accuracy of the evaluation of the target data source.

[0039] S200, periodically probe the target data source through the experimental collection group according to a preset probing period, and obtain a probing result corresponding to each probing according to a crawling result of each experimental crawler on the target data source corresponding to the probing; the probing result corresponding to each probing includes a crawling success number and a crawling success rate; the crawling result of any experimental crawler on the target data source corresponding to each probing is success or failure.

[0040] In this embodiment, the probing period refers to the time interval between two adjacent probings, and the probing period is used to control the probing frequency of the target data source. The preset probing period is a pre-set time parameter; optionally, the preset probing period is an empirical value, for example, 10 minutes / time or 30 minutes / time. In each probing period, the experimental crawler initiates a standardized crawling request to the target data source, and records the crawling result (success / failure).

[0041] In this embodiment, the periodic probing refers to a process of periodically initiating a standardized crawling request to the target data source by the experimental collection group according to the preset probing period. For each probing, the number of experimental crawlers in the experimental collection group corresponding to the probing whose crawling result on the target data source is success is determined as the crawling success number of the target data source; and the ratio of the crawling success number of the target data source to the number of experimental crawlers included in the experimental collection group is determined as the crawling success rate of the target data source.

[0042] In this embodiment, the crawling success number and the success rate of the target data source are used to evaluate the carrying capacity and the crawlable performance of the target data source, and the carrying capacity and the crawlable performance of the target data source are converted into quantifiable indexes, which provides data basis for subsequent determination of the collection power index of the target data source.

[0043] S300, obtain the latest crawling success number and the latest crawling success rate of the target data source according to the probing result corresponding to the latest preset probing.

[0044] In the embodiment, the preset number of times is a preset number parameter, which is used to filter historical obsolete data, focus on the recent state of the data source, and balance timeliness and stability. Optionally, the preset number of times is an experience value, for example, 1, 3 or 5.

[0045] As a specific embodiment, the latest crawling success number B is B = ∑ N j=1 (δ j ×B j ), B j is the crawling success number corresponding to the latest jth detection, δ j is the weight corresponding to the latest jth detection, j ranges from 1 to N, and N is the preset number of times; optionally, δ j = 1 / N. The latest crawling success rate p is p = ∑ N j=1 (δ j ×p j ), p j is the crawling success rate corresponding to the latest jth detection.

[0046] S400, according to the latest crawling success number, the latest crawling success rate and the priority of the target data source, obtaining the collection power index of the target data source.

[0047] In the embodiment, the collection power index of the target data source is an index for comprehensively measuring the priority, the latest crawling success number and the latest crawling success rate of the target data source. For the target data source, the crawling success number of the target data source is positively correlated with the collection power index of the target data source, the crawling success rate of the target data source is negatively correlated with the collection power index of the target data source, and the priority of the target data source is positively correlated with the collection power index of the target data source.

[0048] As a specific embodiment, as shown in Figure 2 , S400 includes:

[0049] S410, determining the ratio of the first dynamic coefficient to the latest crawling success rate of the target data source as the first collection power sub-index of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0050] In the embodiment, the first dynamic coefficient is a dynamic adjustment parameter related to the latest crawling success rate of the target data source. As a specific embodiment, the first dynamic coefficient is positively correlated with the latest crawling success rate of the target data source.

[0051] In this embodiment, the first collection power sub-index reflects the contribution of the success rate to the collection power index, the first collection power sub-index is I1, I1=α / p, and the value range of I1 is [0, 1].

[0052] As a specific implementation, the first dynamic coefficient is α, α=(e b×(p-1) -e -b ) / (1-e -b ), p is the latest crawling success rate of the target data source, and b is a preset parameter value, b>0. Optionally, b is an empirical value, for example, b=2. Based on the formula, α increases with the increase of the latest success rate p of the target data source, when p→0 (the latest success rate of the target data source is extremely low), α≈0, so I1≈0, which can significantly reduce the weight of the target data source, which is consistent with the scene that unstable data source should reduce resources; when p is at a medium level, α grows with p but the growth rate is slower than p, so I1=α / p grows with p but the growth rate is gentle, avoiding that the target data source obtains too high weight at a medium success rate; when p→1 (the success rate is extremely high), α≈1, so I1≈1, which can allocate a larger weight to the target data source, and does not additionally amplify the weight. Based on the formula, non-linear adjustment is realized through an exponential function, the punishment for low success rate data source is more significant (α is much smaller than p), the reward for high success rate data source is more restrained (α is close to p), avoiding that resources are tilted to extremely unstable or excessively compatible data sources.

[0053] In S420, the ratio of the latest crawling success number of the target data source to the second dynamic coefficient is determined as the second collection power sub-index of the target data source; the second collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0054] In this embodiment, the second dynamic coefficient is a dynamic adjustment parameter related to the maximum value of the historical success number of the target data source, as a specific implementation, for the target data source, the second dynamic coefficient is determined according to the maximum value of the crawling success number of the target data source in the target historical time period.

[0055] In this embodiment, the second collection power sub-index reflects the contribution of the success number to the collection power index, the second collection power sub-index is I2, I2=B / β, B is the latest crawling success number of the target data source, and the value range of the second collection power sub-index is [0, 1].

[0056] As a specific implementation, β=w1×B 1,max +(1-w1)×B 2,max . B 1,max is the maximum value of the crawling success number of the target data source in a specified unit time period, and B 2,maxB1,max is the maximum value of the successful crawling number of the target data source in the target historical time period except for the specified unit time period, the specified unit time period is the unit time period closest to the current time; w1 is a preset weight, and 0.5 < w1 < 1. Based on the formula, β integrates the best performance of the recent (B1,max) and the history (B2,max), can avoid the misjudgment caused by the single time period data (such as short-term fluctuation or obsolete data), and w1 > 0.5 ensures that the recent performance weight is higher; β updates with the historical performance of the data source, so that I2 can reflect the position of the current value of the data source relative to the historical best level, rather than the absolute value under the fixed benchmark; the essence of I2 = B / β is the ratio of the current performance to the historical best performance, when B is close to the recent or historical maximum value, I2→1 (indicating that the current performance is excellent); when B is much smaller than the historical maximum value, I2→0 (indicating that the current performance is poor), the embodiment measures the value of the current successful number B through β, which not only pays attention to the recent performance of the data source, but also considers its long-term stability, and finally makes I2 reflect the advantages and disadvantages of the current successful number relative to the historical best level.

[0057] As a specific embodiment, the target historical time period includes a plurality of unit times, the target data source is detected a plurality of times in any unit time, and the target historical time period is 7 days, the unit time period is 1 day, and the target data source is detected once every 30 minutes (that is, the detection period is 30 minutes), and one successful crawling number can be obtained each time.

[0058] In S430, the first acquisition power sub-index, the second acquisition power sub-index and the priority of the target data source are used to obtain the acquisition power index of the target data source; the acquisition power index of the target data source is positively correlated with the first acquisition power sub-index, and the acquisition power index of the target data source is positively correlated with the second acquisition power sub-index.

[0059] In the embodiment, the priority reflects the importance of the data source, the higher the importance, the greater the priority; the value range of the priority is [0, 1]. Optionally, the priority of the target data source is manually set.

[0060] As a specific embodiment, the product of the first acquisition power sub-index, the second acquisition power sub-index and the priority of the target data source is determined as the acquisition power index of the target data source.

[0061] In the embodiment, the first acquisition power sub-index, the second acquisition power sub-index and the priority are used for cooperative evaluation in three dimensions, and finally an index (that is, the acquisition power index) which comprehensively reflects the degree of value of the data source is output, which avoids the single dimension to dominate the resource allocation, improves the comprehensiveness of the evaluation, and makes the allocation of the data acquisition resource more accurate, more dynamic and more in line with the actual demand.

[0062] S500, determine the collection power of the target data source according to the collection power index of the target data source, and collect data of the target data source based on the collection power of the target data source.

[0063] In this embodiment, the collection power index of the target data source is positively correlated with the collection power of the target data source. As a specific implementation, a mapping relationship between the collection power index and the collection power is established in advance. Based on the collection power index of the target data source, the collection power of the target data source can be obtained through the above-mentioned pre-established mapping relationship.

[0064] In this embodiment, the collection power of the target data source reflects the collection intensity of the target data source, and the collection power of the target data source can be a collection frequency.

[0065] In this embodiment, the target data source is periodically detected by a preset detection period, and the latest crawling success number and success rate are calculated based on the latest preset detection result. The state change (such as anti-crawling strategy upgrade, server load capacity fluctuation, temporary failure, etc.) of the data source can be captured in real time. Compared with the lag response of the traditional fixed crawler strategy to the state change, the collection strategy of this embodiment can keep synchronization with the current state of the target data source, and improve the adaptability of data collection to dynamic environment.

[0066] In this embodiment, the collection power index is calculated based on the latest crawling success number (reflecting the current upper limit of the data source) and the latest crawling success rate (reflecting the current anti-crawling intensity), which can accurately match the real-time accessibility of the data source. For data sources in good condition (high success rate and strong load capacity), higher collection power can be allocated to fully utilize the resources; for data sources in poor condition (low success rate and frequent interception), the collection power can be automatically reduced to reduce invalid requests (such as intercepted crawler resources and wasted bandwidth). This mechanism solves the problems of resource waste in poor state and insufficient collection in good state in the traditional fixed strategy, and improves the overall resource utilization efficiency.

[0067] In this embodiment, the priority of the target data source and the latest state index (i.e. the latest crawling success number and the latest crawling success rate) are jointly included in the calculation of the collection power index, so that the high-priority data source can obtain reasonable resource tilt within the range allowed by its current state: when the high-priority data source is in good condition, resources are preferentially allocated to ensure efficient collection of core data; when the high-priority data source is in poor condition, resources are not blindly invested to avoid invalid consumption, but are dynamically adjusted based on the actual accessibility, balancing the priority weight and real-time feasibility, and improving the stability and reliability of core data collection.

[0068] Therefore, the embodiment reduces the dependence on manual intervention through the full-process automatic mechanism of periodic detection, automatic calculation of the latest state indicators, and dynamic generation of collection power, is especially suitable for multi-data source and high-dynamic collection scenarios, and significantly reduces the labor cost.

[0069] Embodiment Two

[0070] Compared with Embodiment One, the embodiment further provides a process of determining the priority of the target data source, and the process includes:

[0071] G100, obtaining a data access heat of the target data source according to a number of times that data of the target data source is accessed within a target time period; the target time period is a time period with a preset length and ending at a current time.

[0072] In the embodiment, the preset length is an empirical value, which can be flexibly set according to a business scenario (such as 1 day, 7 days, etc.). For example, for a scenario with high real-time requirements, 1 day can be set, and for a scenario with low real-time requirements, 7 days can be set.

[0073] In the embodiment, the target data source refers to an object that needs to be collected, and the target data source can generate data. Optionally, the target data source is a website.

[0074] In the embodiment, the data access heat of the target data source refers to the frequency of access to the target data source, and the data access heat of the target data source is positively correlated with the number of times that data of the target data source is accessed within the target time period. As a specific implementation, the number of times that data of the target data source is accessed within the target time period is normalized to obtain the data access heat of the target data source. Optionally, the data access heat of the target data source is H, H=(c-c min ) / (c max -c min ), where c is the number of times that data of the target data source is accessed within the target time period, c min and c max are preset minimum and maximum access times, respectively. Thus, the number of times that data of the target data source is accessed within the target time period can be standardized to the interval [0, 1].

[0075] G200, obtaining a data update activity of the target data source according to an update frequency of the target data source within the target time period.

[0076] In the embodiment, the data update activity of the target data source reflects the dynamic characteristics (average level and change trend) of the data update of the target data source, and the dynamic characteristics of the data update of the target data source directly affect the data timeliness. As a specific implementation, G200 includes:

[0077] G210, obtain an update frequency curve of the target data source in the target time period according to the update frequency of the target data source in the target time period; the update frequency of the target data source in the target time period comprises the update frequency of the target data source corresponding to each unit time in the target time period; the horizontal axis of the update frequency curve of the target data source in the target time period is time, and the vertical axis of the update frequency curve of the target data source in the target time period is the update frequency.

[0078] As a specific embodiment, the update frequency (i.e., the number of data updates) of the target data source in each unit time included in the target time period is counted, and an update frequency curve is drawn with time as the horizontal axis and the update frequency as the vertical axis; based on the update frequency curve, the trend of the update frequency of the target data source over time can be obtained.

[0079] G220, obtain an average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period.

[0080] In this embodiment, the absolute value of the slope of the update frequency curve of the target data source in the target time period can reflect the fluctuation degree of the update frequency, and the average value of the absolute value of the slope can quantify the overall fluctuation trend. The greater the average value of the absolute value of the slope, the more intense the update frequency fluctuation, and the stronger the information capturing ability of the target data source.

[0081] G230, obtain the data update activity of the target data source according to the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period and the average value of the update frequency of the target data source in different unit times in the target time period.

[0082] In this embodiment, the average level of the update frequency (basic activity) and the fluctuation trend (dynamic activity) are combined to comprehensively evaluate the update characteristics of the target data source. As a specific embodiment, G230 comprises:

[0083] G231, normalize the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period to obtain a first activity of the target data source.

[0084] As a specific embodiment, the first activity is A1, A1=(z-z min ) / (z max -z min ), where z is the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period, z min and z max are the minimum value and the maximum value of the average value of the absolute value of the slope, respectively. In this way, the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period can be standardized to the interval [0, 1].

[0085] G232, normalizing the average value of the update frequency of the target data source in different unit time in the target time period to obtain a second activity of the target data source.

[0086] As a specific embodiment, the second activity is A2, A2=(f-f min ) / (f max -f min ), where f is the average value of the update frequency of the target data source in different unit time in the target time period, f min and f max are the minimum and maximum values of the average value of the preset update frequency, respectively. Thus, the average value of the update frequency of the target data source in different unit time in the target time period can be standardized to the interval [0, 1].

[0087] G233, obtaining the data update activity of the target data source according to the weight corresponding to the slope of the preset update frequency curve, the first activity of the target data source, the weight corresponding to the average value of the preset update frequency, and the second activity of the target data source.

[0088] As a specific embodiment, the data update activity of the target data source is A, A=w k ×A1+w f ×A2, w k is the weight corresponding to the slope of the preset update frequency curve, w f is the weight corresponding to the average value of the preset update frequency, w k +w f =1, and w k >0, w f >0. w k reflects the importance of the update frequency fluctuation trend, and w f reflects the importance of the average update level, which can be set according to experience, for example, w k =w f =0.5, or w k =0.4, w f =0.6.

[0089] Thus, the embodiment comprehensively evaluates the data update activity of the target data source by combining dynamic and static update characteristics.

[0090] G300, obtaining the output data amount index of the target data source according to the amount of data generated by the target data source in the target time period.

[0091] In this embodiment, the output data volume index of the target data source reflects the size of the data volume generated by the target data source in the target time period, and the output data volume index of the target data source is positively correlated with the data volume generated by the target data source in the target time period. The greater the output data volume of the target data source, the richer the information contained in the target data source.

[0092] As a specific implementation, the total data volume v (e.g., in bytes) generated by the target data source in the target time period is counted, and a normalized processing is performed to obtain the output data volume index v of the target data source idx , v idx =(v-v min ) / (v max -v min ), where v min and v max are the preset minimum data volume and maximum data volume, respectively. Thus, the data volume generated by the target data source in the target time period can be standardized to the interval [0, 1].

[0093] G400, obtaining the data collection resource consumption index of the target data source according to the resources consumed by the data collection of the target data source in the target time period.

[0094] In this embodiment, the data collection resource consumption index of the target data source reflects how much resource is consumed in collecting the data of the target data source, and the data collection resource consumption index of the target data source is positively correlated with the resources consumed by the data collection of the target data source in the target time period. The more resources consumed in the collection process, the higher the collection cost, which needs to be balanced with resource efficiency in the priority evaluation.

[0095] As a specific implementation, the resources of each preset type consumed in collecting the target data source in the target time period are counted, each preset type of resource is normalized, and the normalized results are weighted and summed to obtain the resource consumption index. Optionally, the preset type of resource includes CPU occupancy, memory occupancy, and bandwidth occupancy, the normalization adopts an existing normalization method, for example, a min-max normalization method; the weight corresponding to each preset type of resource used when the normalized results are weighted and summed is an empirical value, the weight corresponding to each preset type of resource is greater than 0, and the sum of the weights corresponding to each preset type of resource is 1, for example, the weight corresponding to each preset type of resource is 1 / R, and R is the number of preset types of resources.

[0096] G500, obtaining the priority of the target data source according to the data access frequency, the data update activity, the output data volume index, and the data collection resource consumption index of the target data source.

[0097] In this embodiment, the priority of the target data source comprehensively evaluates the importance of the target data source. As a specific implementation, G500 includes:

[0098] G510 obtains the weights corresponding to data access popularity, data update activity, output data volume, and data collection resource consumption; the weights corresponding to data access popularity, data update activity, output data volume, and data collection resource consumption are all greater than 0.

[0099] As a specific implementation method, the weights corresponding to data access popularity, data update activity, data output volume, and data collection resource consumption include:

[0100] G511, obtain the target variable; the target variable is the manually labeled priority.

[0101] As a specific implementation method, the priority of manually labeled data is collected as the target variable Y (the historical data source is labeled by domain experts according to business needs, and the value range is [0,1]).

[0102] G512 uses four indicators from the target data source as feature variables to construct a dataset containing the correspondence between feature variables and target variables; the four indicators include data access popularity, data update activity, output data volume index, and data collection resource efficiency.

[0103] As a specific implementation method, a dataset is constructed, where each data point includes a feature variable and a corresponding target variable. The data acquisition resource efficiency is defined as the difference between 1 and the data acquisition resource consumption index.

[0104] G513, a linear regression model is trained based on the dataset to obtain the coefficients corresponding to each feature variable in the linear regression model.

[0105] As a specific implementation method, a linear regression model is trained, with the model expression being Y=w h ×x1+w a ×x2+w v ×x3+w r ×x4, w h w represents the characteristic coefficient corresponding to the data access popularity. a w represents the feature coefficient corresponding to the data update activity. v w represents the characteristic coefficient corresponding to the output data volume index. r Let x1, x2, x3, and x4 be the feature coefficients corresponding to the data acquisition resource consumption index, respectively. x1, x2, x3, and x4 are the feature variables corresponding to data access popularity, data update activity, output data volume index, and data acquisition resource efficiency, respectively. During training, the coefficients for w are added... h w a wv and the non-negative constraint of w r , guarantees w h , w a , w v and w r are positive; the linear regression model is trained by least square method. Those skilled in the art know that the specific training process is prior art, which is not described here.

[0106] G514, normalize the coefficients, so that the sum of the normalized coefficients is 1 and each coefficient is greater than 0, and the normalized coefficients are respectively determined as the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume and the weight corresponding to the data collection resource consumption.

[0107] As a specific embodiment, the weight corresponding to the data access heat is w h , w h = w h / (w h +w a +w v +w r ); the weight corresponding to the data update activity is w a , w a = w a / (w h +w a +w v +w r ); the weight corresponding to the output data volume is w v , w v = w v / (w h +w a +w v +w r ); and the weight corresponding to the data collection resource consumption is w r , w r = w r / (w h +w a +w v +w r ).

[0108] Based on G511-G514, the embodiment determines the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume and the weight corresponding to the data collection resource consumption in a data-driven manner, and the weights thus determined can be more in line with actual business scenarios.

[0109] G520, determines the product of the weight corresponding to the data access heat and the data access heat of the target data source as the first sub-priority of the target data source.

[0110] G530, determining the product of the weight corresponding to the data update activity and the data update activity of the target data source as the second sub-priority of the target data source.

[0111] G540, determining the product of the weight corresponding to the output data volume and the output data volume index of the target data source as the third sub-priority of the target data source.

[0112] G550, determining the product of the weight corresponding to the data acquisition resource consumption and the data acquisition resource efficiency of the target data source as the fourth sub-priority of the target data source; the data acquisition resource efficiency of the target data source is the difference between 1 and the data acquisition resource consumption index of the target data source.

[0113] G560, determining the sum of the first sub-priority of the target data source, the second sub-priority of the target data source, the third sub-priority of the target data source and the fourth sub-priority of the target data source as the priority of the target data source.

[0114] Based on G520-G560, the priority of the target data source is obtained by weighted fusion of the data access heat, the data update activity, the output data volume index and the data acquisition resource efficiency of the target data source, so that the priority of the target data source determined considers more comprehensive dimensions.

[0115] Embodiment three:

[0116] Compared with embodiment one, the embodiment further provides a process of determining the acquisition power of the target data source according to the acquisition power index of the target data source, which comprises:

[0117] S510, obtaining a set of experimental crawlers whose crawling results on the target data source are successful in an experimental acquisition group; the experimental acquisition group includes a plurality of experimental crawlers.

[0118] In this embodiment, the experimental acquisition group is a set composed of a plurality of experimental crawlers, and the experimental acquisition group includes different experimental crawlers. The experimental acquisition group is used for preliminary crawling test on the candidate data sources in the candidate data source set, and the crawling results are obtained to evaluate the data source performance. The candidate data source set is a set of data sources (such as websites) to be evaluated, which is the target of data acquisition. The experimental crawler is a program used to simulate the data acquisition process, which can initiate a crawling request to the data source and return a success or failure result. The candidate data source is a specific data source to be evaluated, and the target data source is any candidate data source in the candidate data source set, which is the object of evaluation in this embodiment. As a specific embodiment, the target data source and the candidate data source are websites.

[0119] In this embodiment, the experimental crawler program and the candidate data source in the experimental collection group meet the preset condition, and the preset condition includes that the experimental crawler program and the candidate data source are compatible. The preset condition that the experimental crawler program and the candidate data source are compatible is a basic prerequisite for the experiment to be effectively carried out. If the crawler and the data source are not compatible, the crawler cannot normally crawl, a large amount of invalid failure data is generated, the set of experimental crawler programs that successfully crawl cannot be filtered out, and the entire experimental process is stalled. Based on the preset condition that the experimental crawler program and the candidate data source are compatible, it can be ensured that all experimental crawlers can effectively access and crawl the candidate data source, and invalid samples are avoided.

[0120] In this embodiment, the preset data source attributes of the candidate data sources in the candidate data source set are the same, and the preset data source attributes include at least one of the following attributes: data source type, anti-crawling strategy, page structure type, access protocol, and request frequency limit value. The data source type includes static website, dynamic website, etc., the anti-crawling strategy includes verification code verification, IP restriction, User-Agent detection, etc., the page structure type includes list type structure, detail type structure, nested type structure, irregular structure, etc., the access protocol includes HTTP, FTP, HTTPS, etc., and the request frequency limit value refers to the maximum number of requests allowed by the data source within a unit time. Thus, the success rate of the experimental crawler program can be prevented from being distorted due to the attribute difference of the data source, the success rate of the experimental crawler program can truly reflect the characteristics of the data source itself, and the data sample without bias is improved for subsequent determination of the crawling stability of the target data source based on the success rate of the experimental crawler program.

[0121] In this embodiment, the preset crawler program attributes of different experimental crawler programs in the experimental collection group are the same, and the preset crawler program attributes include at least one of the following attributes: crawling target, running parameter, and validity. The crawling target refers to specific data content that needs to be obtained from the data source, for example, title, publication time, and text of a news website, etc. The running parameter refers to the configuration parameter of the crawler program, including the number of concurrent connections, timeout time, number of retries, etc. The validity refers to the functional integrity and correctness of the crawler program itself. The preset attributes of the experimental crawler program are the same, which can exclude the influence of the differences of the crawlers on the experimental results, make the crawling ability and execution logic of all experimental crawler programs consistent, and make the difference in the success rate determined by the real characteristics of the candidate data source, so that the crawling characteristic determination index calculated based on the success rate is more objective, and truly reflects the stability of the target data source.

[0122] In this embodiment, each experimental crawler in the experimental collection set is used to perform crawling test on each candidate data source, so that the crawling result of each experimental crawler in the experimental collection set on each candidate data source can be obtained, and the crawling result includes success and failure. The set of experimental crawlers whose crawling result on the target data source is success is determined as the experimental crawler set.

[0123] In S520, the crawling success rate of each experimental crawler in the experimental crawler set on the candidate data source in the candidate data source set is obtained; the candidate data source set includes a plurality of candidate data sources.

[0124] In this embodiment, the crawling result of each experimental crawler in the experimental crawler set on each candidate data source in the candidate data source set is known, and on this basis, the crawling success rate of each experimental crawler in the experimental crawler set on the candidate data source in the candidate data source set can be obtained. The crawling success rate of any experimental crawler in the experimental crawler set on the candidate data source in the candidate data source set is the ratio of the number of successful crawls of the experimental crawler on the candidate data source in the candidate data source set to the number of candidate data sources included in the candidate data source set.

[0125] In S530, the crawling feature determination index of the target data source is determined according to the crawling success rate.

[0126] As a specific implementation, S530 includes:

[0127] In S531, the average value of the crawling success rate of the experimental crawler in the experimental crawler set on the candidate data source in the candidate data source set is obtained.

[0128] In S532, the variance of the crawling success rate of the experimental crawler in the experimental crawler set on the candidate data source in the candidate data source set is obtained.

[0129] In S533, the crawling feature determination index of the target data source is determined according to the average value and the variance of the crawling success rate; the crawling feature determination index of the target data source is negatively correlated with the average value of the crawling success rate; and the crawling feature determination index of the target data source is negatively correlated with the variance of the crawling success rate.

[0130] As a specific implementation, the average value and the variance of the crawling success rate are normalized respectively, then the difference between 1 and the normalized result of the average value is determined as a first index, and the difference between 1 and the normalized result of the variance is determined as a second index; the product of the first index and the second index is determined as the crawling feature determination index of the target data source. Those skilled in the art know that normalization is a prior art, which will not be described here.

[0131] In this embodiment, the greater the crawling feature judgment index of the target data source, the easier the target data source is to be crawled and the more stable the crawling difficulty is; otherwise, the more difficult the target data source is to be crawled or the more unstable the crawling difficulty is.

[0132] S540, if the crawling feature judgment index of the target data source is greater than or equal to a preset judgment index threshold, determining the collection power of the target data source according to a comparison result of the collection power index of the target data source and a preset collection power index threshold; otherwise, determining the collection power of the target data source according to a comparison result of the collection power index of the target data source and an updated collection power index threshold; the updated collection power index threshold is obtained according to the preset collection power index threshold, and the updated collection power index threshold is not equal to the preset collection power index threshold.

[0133] As a specific embodiment, the collection power index of the target data source is a comprehensive index quantitatively describing the resource input intensity and the strategy aggressiveness when collecting data of the target data source, and the range of the collection power index of the target data source is [0, 1], which can be manually set.

[0134] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold includes: if the collection power index of the target data source is greater than or equal to a preset first collection power index threshold, determining the collection power of the target data source as a preset maximum collection power; if the collection power index of the target data source is less than or equal to a preset second collection power index threshold, determining the collection power of the target data source as a preset minimum collection power; the preset first collection power index threshold is greater than the preset second collection power index threshold.

[0135] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold further includes: if the collection power index of the target data source is less than the preset first collection power index threshold and greater than the preset second collection power index threshold, performing normalization processing on the collection power index of the target data source according to the preset first collection power index threshold and the preset second collection power index threshold, and determining the product of the result of the normalization processing on the collection power index of the target data source and the preset maximum collection power as the collection power of the target data source. It should be understood that the normalization processing here refers to maximum-minimum normalization processing.

[0136] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value includes: if the collection power index of the target data source is greater than or equal to the updated first collection power index threshold value, determining the collection power of the target data source as the preset maximum collection power; if the collection power index of the target data source is less than or equal to the updated second collection power index threshold value, determining the collection power of the target data source as the preset minimum collection power; the updated first collection power index threshold value is greater than the updated second collection power index threshold value; the updated first collection power index threshold value is less than the preset first collection power index threshold value, and the updated second collection power index threshold value is less than the preset second collection power index threshold value.

[0137] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value further includes: if the collection power index of the target data source is less than the updated first collection power index threshold value and greater than the updated second collection power index threshold value, performing normalization processing on the collection power index of the target data source according to the updated first collection power index threshold value and the updated second collection power index threshold value, and determining the larger value of the product of the result of the normalization processing on the collection power index of the target data source and the preset maximum collection power and the original collection power as the collection power of the target data source; wherein the original collection power is: the product of the result of the normalization processing on the collection power index of the target data source based on the preset first collection power index threshold value and the preset second collection power index threshold value and the preset maximum collection power. It should be understood that the normalization processing here refers to maximum-minimum normalization processing.

[0138] As a specific embodiment, T'1 is an updated first collection power index threshold, 0 < T'1 < 1, T'1 = (1 - k1 x (k - k0) / k) x T1, where T1 is a preset first collection power index threshold, 0 < T1 < 1, k1 is a preset first adjustment coefficient, 0 < k1 < 1, k is a preset determination index threshold, k0 is a crawling characteristic determination index of the target data source, and k0 < k. Optionally, T1, k1, and k are empirical values, for example, T1 = 0.8, k1 = 0.5, and k = 0.6. Based on the formula, when the actual crawling characteristic determination index of the target data source is less than the preset determination index threshold, the updated first collection power index threshold is less than the preset first collection power index threshold; and the more the actual crawling characteristic determination index of the target data source is less than the preset determination index threshold, the more the updated first collection power index threshold is less than the preset first collection power index threshold, which helps to enable the strong mode earlier to cope with possible crawling difficulties when the data source is unstable or difficult to be crawled.

[0139] As a specific embodiment, T'2 is an updated second collection power index threshold, 0 < T'2 < 1, T'2 = (1 - k2 x (k - k0) / k) x T2, where T2 is a preset second collection power index threshold, 0 < T2 < 1, k2 is a preset second adjustment coefficient, 0 < k2 < 1, k is a preset determination index threshold, k0 is a crawling characteristic determination index of the target data source, k0 < k, and (1 - k1) x T1 > (1 - k2) x T2. Optionally, T2 and k2 are empirical values, for example, T2 = 0.3 and k2 = 0.3. Based on the formula, the collection power index is reduced to a lower level before triggering the minimum power collection instruction, which forces the system to adopt a strategy of increasing the sampling power when the data source is unstable or difficult to be crawled, and is less likely to enter a low-power or sleep state, thereby giving priority to ensuring crawling success.

[0140] The embodiment obtains the crawling success rate of each experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set, determines the crawling characteristic judgment index of the target data source according to the crawling success rate, and if the crawling characteristic judgment index of the target data source is greater than or equal to the preset judgment index threshold value, determines the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold value; otherwise, determines the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value. Thus, the embodiment can intelligently distinguish the stability characteristics of the data source by introducing the crawling characteristic judgment index and constructing a dynamic threshold adjustment mechanism, and for the data source with stable characteristics, a conventional threshold is used for decision-making; for the data source with unstable characteristics, an updated threshold is used for decision-making; such differentiated processing method is conducive to improving the accuracy of power decision-making, thereby improving the success rate and efficiency of data collection as a whole.

[0141] Embodiment four

[0142] The embodiment provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0143] An experimental collection group is obtained; the experimental collection group comprises a plurality of experimental crawler programs.

[0144] According to a preset detection period, the target data source is periodically detected by the experimental collection group, and a detection result corresponding to each detection is obtained according to the crawling result of each experimental crawler program on the target data source corresponding to the detection; the detection result corresponding to each detection comprises a crawling success number and a crawling success rate; the crawling result of any experimental crawler program on the target data source corresponding to each detection is success or failure.

[0145] The latest crawling success number and the latest crawling success rate of the target data source are obtained according to the detection result corresponding to the latest preset detection.

[0146] The collection power index of the target data source is obtained according to the latest crawling success number, the latest crawling success rate and the priority of the target data source.

[0147] The collection power of the target data source is determined according to the collection power index of the target data source, and the target data source is collected based on the collection power of the target data source.

[0148] Embodiment five

[0149] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0150] obtain an experimental collection group; the experimental collection group includes a plurality of experimental crawlers.

[0151] periodically detect the target data source through the experimental collection group according to a preset detection period, and obtain a detection result corresponding to each detection according to a crawling result of each experimental crawler on the target data source; the detection result corresponding to each detection includes a successful crawling number and a successful crawling rate; the crawling result of any experimental crawler on the target data source corresponding to each detection is success or failure.

[0152] obtain a latest successful crawling number and a latest successful crawling rate of the target data source according to a detection result corresponding to a latest preset detection.

[0153] obtain a collection power index of the target data source according to the latest successful crawling number, the latest successful crawling rate and the priority of the target data source.

[0154] determine a collection power of the target data source according to the collection power index of the target data source, and collect data of the target data source based on the collection power of the target data source.

[0155] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0156] While certain specific embodiments of the application have been described in detail herein for the purposes of exemplification, numerous other variations and modifications will be apparent to persons skilled in the art. Alterations and modifications of detail can be made by those skilled in the art, having the benefit of the above description, without departing from the spirit and scope of the application. It is intended that all such alterations and modifications be included within the scope of the application whose limits are to be determined only by the appended claims.

Claims

1. A method of processing data acquisition, characterized by, The method comprises the following steps: acquiring an experimental collection group; the experimental collection group comprises a plurality of experimental crawlers; periodically detecting a target data source by the experimental collection group according to a preset detection period, and acquiring a detection result corresponding to each detection according to a crawling result of each experimental crawler on the target data source in the corresponding detection; the detection result corresponding to each detection comprises a successful crawling number and a successful crawling rate; the crawling result of any experimental crawler on the target data source in each detection is success or failure; acquiring a latest successful crawling number and a latest successful crawling rate of the target data source according to the detection result corresponding to the latest preset detection; acquiring a collection power index of the target data source according to the latest successful crawling number, the latest successful crawling rate and the priority of the target data source; determining a collection power of the target data source according to the collection power index of the target data source, and collecting data of the target data source based on the collection power of the target data source.

2. The processing method of data acquisition according to claim 1, characterized in that, The acquiring of the detection result corresponding to each detection according to the crawling result of each experimental crawler on the target data source comprises: for each detection, determining a successful crawling number of the target data source as a number of experimental crawlers with successful crawling results on the target data source in the experimental collection group corresponding to the detection; and determining a successful crawling rate of the target data source as a ratio of the successful crawling number of the target data source to a number of experimental crawlers included in the experimental collection group.

3. The method of claim 1, wherein, The successful crawling number of the target data source is positively correlated with the collection power index of the target data source, the successful crawling rate of the target data source is negatively correlated with the collection power index of the target data source, and the priority of the target data source is positively correlated with the collection power index of the target data source.

4. The method of claim 3, wherein, The acquiring of the collection power index of the target data source according to the latest successful crawling number, the latest successful crawling rate and the priority of the target data source comprises: determining a first collection power sub-index of the target data source as a ratio of a first dynamic coefficient to the latest successful crawling rate of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1; determining a second collection power sub-index of the target data source as a ratio of the latest successful crawling number of the target data source to a second dynamic coefficient; the second collection power sub-index is greater than or equal to 0 and less than or equal to 1; acquiring the collection power index of the target data source according to the first collection power sub-index, the second collection power sub-index and the priority of the target data source; the collection power index of the target data source is positively correlated with the first collection power sub-index, and the collection power index of the target data source is positively correlated with the second collection power sub-index.

5. The method of claim 4, wherein, The first dynamic coefficient is positively correlated with the latest successful crawling rate of the target data source.

6. The method of claim 4, wherein, The second dynamic coefficient is determined according to a maximum value of a successful crawling number of the target data source in a target historical time period.

7. The method of claim 4, wherein, The collection power index of the target data source is determined as a product of the first collection power sub-index, the second collection power sub-index and the priority of the target data source.

8. The method of claim 1, wherein, The collection power index of the target data source is positively correlated with the collection power of the target data source.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the processing method for data collection of any one of claims 1-8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program, when executed by a processor, implements the processing method of data acquisition according to any one of claims 1 to 8.

Images