Data acquisition processing method, equipment and medium

Abstract

The invention relates to the technical field of electric digital data processing, in particular to a data acquisition processing method and device and a medium. The method comprises the following steps: acquiring an experiment acquisition group; according to a preset detection period, periodically detecting a target data source through the experiment acquisition group, and obtaining a detection result corresponding to any detection according to a crawling result of each experiment crawler program corresponding to the detection on the target data source; obtaining the latest crawling success number and the latest crawling success rate of the target data source according to the detection result corresponding to the latest preset detection; acquiring an acquisition power index of the target data source according to the latest crawling success number, the latest crawling success rate and the priority of the target data source; and determining the collection power of the target data source according to the collection power index of the target data source, and carrying out data collection on the target data source based on the collection power of the target data source. According to the invention, dynamic matching of the acquisition power of the data source and the state of the data source is realized.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a data acquisition processing method, device and medium. Background Technology

[0002] With the deep application of big data technology in various fields, data has become a core asset for enterprise decision-making and business optimization, and efficient data collection is a prerequisite for realizing the value of big data. Web crawlers, as core tools for automated data collection, are widely used to obtain target data from various web data sources (such as websites). For example, Chinese patent application number 202510450850.X discloses an adaptive data collection method and apparatus that can automatically adjust the basic crawler program according to different web pages, generating customized crawlers for each web page, thereby improving the accuracy and efficiency of crawling.

[0003] While the aforementioned patent applications can match crawlers and formulate corresponding crawling strategies for different web pages (i.e., data sources), they fail to consider the fact that the state of the web pages themselves can easily change, rendering previously formulated crawling strategies inapplicable to the current web page. It is evident that traditional data collection typically cannot perceive changes in the state of the data source in real time (such as concurrent request handling capacity, anti-crawler strategy upgrades, etc.), leading to the blind allocation of resources even when the data source is unavailable or slow to respond, resulting in a waste of bandwidth and computing resources. How to achieve dynamic matching between the data source collection frequency and the data source state is a problem that urgently needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a data acquisition processing method, device, and medium to achieve dynamic matching between the acquisition frequency of the data source and the status of the data source.

[0005] According to a first aspect of the present invention, a data acquisition and processing method is provided, the method comprising the following steps: Obtain the experimental data collection group; the experimental data collection group includes several experimental web crawler programs.

[0006] According to the preset detection cycle, the experimental acquisition group periodically probes the target data source, and obtains the detection result corresponding to each detection based on the crawling result of each experimental crawler program on the target data source for each detection; the detection result corresponding to each detection includes the number of successful crawls and the crawling success rate; the crawling result of any experimental crawler program on the target data source for each detection is either successful or unsuccessful.

[0007] Based on the detection results of the most recent preset detection, obtain the latest number of successful crawls and the latest crawl success rate of the target data source.

[0008] The collection power index of the target data source is obtained based on the latest number of successful crawls, the latest success rate of crawling, and the priority of the target data source.

[0009] The acquisition power of the target data source is determined based on the acquisition power index of the target data source, and data is acquired from the target data source based on the acquisition power of the target data source.

[0010] Furthermore, the detection results corresponding to each detection are obtained based on the crawling results of each experimental crawler program on the target data source for each detection, including: For any given probe, the number of experimental crawler programs in the experimental collection group corresponding to that probe that successfully crawled the target data source is defined as the number of successful crawls of the target data source; the ratio of the number of successful crawls of the target data source to the number of experimental crawler programs included in the experimental collection group is defined as the success rate of crawling the target data source.

[0011] Furthermore, the number of successful crawls of the target data source is positively correlated with the collection power index of the target data source, the crawl success rate of the target data source is negatively correlated with the collection power index of the target data source, and the priority of the target data source is positively correlated with the collection power index of the target data source.

[0012] Furthermore, the collection power index of the target data source is obtained based on the latest successful crawl count, latest success rate, and priority. The ratio of the first dynamic coefficient to the latest crawling success rate of the target data source is determined as the first collection power sub-index of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0013] The ratio of the latest successful crawl count of the target data source to the second dynamic coefficient is determined as the second acquisition power sub-index of the target data source; the second acquisition power sub-index is greater than or equal to 0 and less than or equal to 1.

[0014] The acquisition power index of the target data source is obtained based on the first acquisition power sub-index, the second acquisition power sub-index, and the priority of the target data source; the acquisition power index of the target data source is positively correlated with the first acquisition power sub-index, and the acquisition power index of the target data source is positively correlated with the second acquisition power sub-index.

[0015] Furthermore, the first dynamic coefficient is positively correlated with the latest crawling success rate of the target data source.

[0016] Furthermore, the second dynamic coefficient is determined based on the maximum number of successful crawls of the target data source during the target historical time period.

[0017] Furthermore, the product of the first acquisition power sub-index, the second acquisition power sub-index, and the priority of the target data source is determined as the acquisition power index of the target data source.

[0018] Furthermore, the acquisition power index of the target data source is positively correlated with the acquisition power of the target data source.

[0019] According to a second aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data acquisition processing method.

[0020] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described data acquisition processing method.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects: This invention periodically probes the target data source through a preset probe cycle and calculates the latest successful crawling count and success rate based on the results of the most recent preset probes. It can capture real-time changes in the data source's status (such as anti-crawling policy upgrades, server capacity fluctuations, and temporary failures). Compared to the lagging response of traditional fixed crawler strategies to status changes, this invention keeps the collection strategy synchronized with the current state of the target data source, improving the data collection's adaptability to dynamic environments.

[0022] This invention calculates a data collection power index based on the latest number of successful crawls (reflecting the current capacity limit of the data source) and the latest crawl success rate (reflecting the current anti-crawling strength), which can accurately match the real-time accessibility of the data source: for data sources in good condition (high success rate, strong capacity), higher collection power can be allocated to make full use of their resources; for data sources in poor condition (low success rate, frequent blocking), the collection power is automatically reduced to reduce invalid requests (such as blocked crawler resources and wasted bandwidth). This mechanism solves the problems of resource waste when the condition is poor and insufficient collection when the condition is good in traditional fixed strategies, thus improving the overall resource utilization efficiency.

[0023] This invention incorporates the target data source priority and the latest status indicators (i.e., the latest number of successful crawls and the latest crawl success rate) into the calculation of the collection power index, ensuring that high-priority data sources receive reasonable resource allocation within the limits allowed by their current status: when the status of a high-priority data source is good, resources are allocated first to ensure the efficient collection of core data; when the status of a high-priority data source deteriorates, ineffective consumption caused by blindly investing resources is avoided, and instead, dynamic adjustments are made based on its actual accessibility, balancing priority weight and real-time feasibility, and improving the stability and reliability of core data collection.

[0024] Therefore, this invention reduces the reliance on manual intervention through a fully automated mechanism that periodically detects, automatically calculates the latest status indicators, and dynamically generates acquisition power. It is especially suitable for acquisition scenarios with multiple data sources and high dynamics, and significantly reduces labor costs. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of the data acquisition and processing method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating the steps for obtaining the acquisition power index of a target data source, as provided in Embodiment 1 of the present invention. Detailed Implementation

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] Example 1: According to this embodiment, as Figure 1 As shown, a data acquisition and processing method is provided, the method including the following steps: S100, Obtain the experimental data collection group; the experimental data collection group includes several experimental crawler programs.

[0029] In this embodiment, the experimental data collection group is a collection of several experimental web crawler programs. The experimental data collection group includes different experimental web crawler programs and is used to perform preliminary crawling tests on the target data source, obtaining crawling results to evaluate the performance of the target data source. The experimental web crawler programs are programs used to simulate the data collection process; they can initiate crawling requests to the target data source and return success or failure results.

[0030] The target data source is the data source to be evaluated (such as a website), which is the target of data collection.

[0031] This embodiment tests the target data source by crawling different experimental crawler programs, which can improve the accuracy of the evaluation of the target data source.

[0032] S200, according to the preset detection cycle, the experimental acquisition group periodically detects the target data source, and obtains the detection result corresponding to the detection based on the crawling result of each experimental crawler program corresponding to the target data source for any detection; the detection result corresponding to any detection includes the number of successful crawls and the crawling success rate; the crawling result of any experimental crawler program corresponding to any detection for the target data source is either successful or unsuccessful.

[0033] In this embodiment, the detection period refers to the time interval between two adjacent detections. The detection period is used to control the detection frequency of the target data source. The preset detection period is a pre-set time parameter; optionally, the preset detection period is an empirical value, such as 10 minutes / time or 30 minutes / time. Within each detection period, the experimental crawler program initiates standardized crawling requests to the target data source and records the crawling results (success / failure).

[0034] In this embodiment, periodic probing refers to the process by which the experimental acquisition group periodically initiates standardized crawling requests to the target data source according to a preset probing cycle. For any given probing, the number of experimental crawler programs in the experimental acquisition group that successfully crawl the target data source is determined as the number of successful crawls of the target data source; the ratio of the number of successful crawls of the target data source to the number of experimental crawler programs included in the experimental acquisition group is determined as the crawling success rate of the target data source.

[0035] In this embodiment, the carrying capacity and crawlability of the target data source are evaluated based on the number of successful crawls and the success rate of the target data source. The carrying capacity and crawlability of the target data source are converted into quantifiable indicators, providing a data basis for subsequently determining the collection power index of the target data source.

[0036] S300: Obtain the latest number of successful crawls and the latest success rate of crawling the target data source based on the detection results corresponding to the most recent preset detection.

[0037] In this embodiment, the preset number refers to the number of recent probe results referenced when calculating the latest successful crawl count and the latest success rate. It is a pre-set quantity parameter. The preset number is used to filter out historical outdated data, focus on the recent state of the data source, and balance timeliness and stability. Optionally, the preset number is an empirical value, for example, 1, 3, or 5.

[0038] As a specific implementation method, the latest number of successful crawls is B, where B = ∑ N j=1 (δ j ×B j ), B j Let δ be the number of successful crawls corresponding to the most recent j-th probe. jLet δ be the weight corresponding to the most recent j-th probe, where j ranges from 1 to N, and N is a preset number of probes; optionally, δ j =1 / N. The latest crawling success rate is p, p=∑ N j=1 (δ j ×p j ), p j This represents the crawling success rate corresponding to the most recent j-th probe.

[0039] S400 obtains the collection power index of the target data source based on the latest successful crawling count, latest success rate, and priority of the target data source.

[0040] In this embodiment, the collection power index of the target data source is an indicator that comprehensively measures the priority of the target data source, the latest number of successful crawls, and the latest crawl success rate. For the target data source, the number of successful crawls is positively correlated with the collection power index, the crawl success rate is negatively correlated with the collection power index, and the priority of the target data source is positively correlated with the collection power index.

[0041] As a specific implementation method, such as Figure 2 As shown, S400 includes: S410, the ratio of the first dynamic coefficient to the latest crawling success rate of the target data source is determined as the first collection power sub-index of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0042] In this embodiment, the first dynamic coefficient is a dynamically adjusted parameter related to the latest crawling success rate of the target data source. As a specific implementation, the first dynamic coefficient is positively correlated with the latest crawling success rate of the target data source.

[0043] In this embodiment, the first acquisition power sub-index reflects the contribution of the success rate to the acquisition power index. The first acquisition power sub-index is I1, I1=α / p, and the value range of I1 is [0,1].

[0044] As a specific implementation method, the first dynamic coefficient is α, α=(e b×(p-1) -e -b ) / (1-e -bLet p be the latest crawling success rate of the target data source, and b be a preset parameter value, where b > 0. Optionally, b can be an empirical value, for example, b = 2. Based on this formula, α increases as the latest success rate p of the target data source increases. When p → 0 (the latest success rate of the target data source is extremely low), α ≈ 0, therefore I1 ≈ 0, which can significantly reduce the weight of the target data source, consistent with the scenario where resources should be reduced for unstable data sources. When p is at a moderate level, α increases with p but at a slower rate than p, therefore I1 = α / p increases with p but at a slower rate, preventing the target data source from receiving excessively high weight at a moderate success rate. When p → 1 (the success rate is extremely high), α ≈ 1, therefore I1 ≈ 1, which can allocate a larger weight to the target data source without further amplifying its weight. Based on this formula, a non-linear adjustment is achieved through an exponential function, resulting in a more significant penalty for low-success-rate data sources (α is much smaller than p) and a more restrained reward for high-success-rate data sources (α is close to p), preventing resources from being tilted towards extremely unstable or overly compatible data sources.

[0045] S420, the ratio of the latest successful crawling count of the target data source to the second dynamic coefficient is determined as the second acquisition power sub-index of the target data source; the second acquisition power sub-index is greater than or equal to 0 and less than or equal to 1.

[0046] In this embodiment, the second dynamic coefficient is a dynamically adjusted parameter related to the maximum number of successful crawls in the target data source's history. As a specific implementation, for the target data source, the second dynamic coefficient is determined based on the maximum number of successful crawls in the target data source's historical time period.

[0047] In this embodiment, the second acquisition power sub-index reflects the contribution of the number of successful crawls to the acquisition power index. The second acquisition power sub-index is I2, I2=B / β, where B is the latest successful crawling count of the target data source, and the value range of the second acquisition power sub-index is [0,1].

[0048] As a specific implementation, β = w1 × B 1,max +(1-w1)×B 2,max B 1,max B represents the maximum number of successful crawls of the target data source within a specified time period. 2,maxis the maximum number of successful crawls of the target data source in other unit time periods within the target historical time period except for the specified unit time period, and the specified unit time period is the unit time period closest to the current moment; w1 is a preset weight, and optionally, 0.5 < w1 < 1. Based on this formula, β integrates the best performances in the recent (B1,max) and historical (B2,max) periods, which can avoid misjudgments caused by data in a single time period (such as short-term fluctuations or stale data), and w1 > 0.5 ensures that the weight of recent performance is higher; β is updated according to the historical performance of the data source, enabling I2 to reflect in real time the position of the current value of the data source relative to its own historical best level, rather than an absolute value under a fixed benchmark; the essence of I2 = B / β is the ratio of the current performance of the data source to its historical best performance. When B approaches the recent or historical maximum value, I2 → 1 (indicating excellent current performance); when B is much smaller than the historical maximum value, I2 → 0 (indicating poor current performance). In this embodiment, β is used to measure the value of the current success count B, which not only pays attention to the recent performance of the data source but also takes into account its long-term stability, and finally enables I2 to reflect the superiority or inferiority of the current success count relative to the historical best level.

[0049] As a specific implementation, the target historical time period includes multiple unit times. In any unit time, the target data source is periodically detected multiple times. Optionally, the target historical time period is 7 days, the unit time period is 1 day, and the target data source is detected once every 30 minutes (i.e., the detection period is 30 minutes), and a successful crawl count can be obtained for each detection.

[0050] S430, obtain the acquisition power index of the target data source according to the first acquisition power sub-index, the second acquisition power sub-index, and the priority of the target data source; the acquisition power index of the target data source is positively correlated with the first acquisition power sub-index, and the acquisition power index of the target data source is positively correlated with the second acquisition power sub-index.

[0051] In this embodiment, the priority reflects the importance of the data source. The higher the importance, the greater the priority; the value range of the priority is [0, 1]. Optionally, the priority of the target data source is set manually.

[0052] As a specific implementation, the product of the first acquisition power sub-index, the second acquisition power sub-index, and the priority of the target data source is determined as the acquisition power index of the target data source.

[0053] In this embodiment, through the coordinated evaluation of the three dimensions of the first acquisition power sub-index, the second acquisition power sub-index, and the priority, an index that comprehensively reflects the degree to which the data source is worthy of being acquired (i.e., the acquisition power index) is finally output, avoiding the dominance of a single dimension in resource allocation, improving the comprehensiveness of the evaluation, and making the allocation of data acquisition resources more accurate, more dynamic, and more in line with actual needs.

[0054] S500 determines the acquisition power of the target data source based on the acquisition power index of the target data source, and acquires data from the target data source based on the acquisition power of the target data source.

[0055] In this embodiment, the acquisition power index of the target data source is positively correlated with the acquisition power of the target data source. As a specific implementation, a mapping relationship between the acquisition power index and the acquisition power is established in advance. Knowing the acquisition power index of the target data source, the acquisition power of the target data source can be obtained through the aforementioned pre-established mapping relationship.

[0056] In this embodiment, the acquisition power of the target data source reflects the acquisition intensity of the target data source, and the acquisition power of the target data source can be the acquisition frequency.

[0057] This embodiment periodically probes the target data source through a preset probe cycle and calculates the latest successful crawling count and success rate based on the results of the most recent preset probes. It can capture changes in the data source's status in real time (such as anti-crawling policy upgrades, server capacity fluctuations, temporary failures, etc.). Compared to the lagging response of traditional fixed crawler strategies to status changes, this embodiment can keep the collection strategy synchronized with the current status of the target data source, improving the data collection's adaptability to dynamic environments.

[0058] This embodiment calculates the collection power index based on the latest number of successful crawls (reflecting the current capacity limit of the data source) and the latest crawl success rate (reflecting the current anti-crawling strength), which can accurately match the real-time accessibility of the data source: for data sources in good condition (high success rate, strong capacity), higher collection power can be allocated to make full use of their resources; for data sources in poor condition (low success rate, frequent blocking), the collection power is automatically reduced to reduce invalid requests (such as blocked crawler resources, wasted bandwidth). This mechanism solves the problem of resource waste when the condition is poor and insufficient collection when the condition is good in traditional fixed strategies, thus improving the overall resource utilization efficiency.

[0059] This embodiment incorporates the target data source priority and the latest status indicators (i.e., the latest number of successful crawls and the latest crawl success rate) into the calculation of the collection power index to ensure that high-priority data sources receive reasonable resource allocation within the allowable range of their current status: when the status of a high-priority data source is good, resources are allocated first to ensure the efficient collection of core data; when the status of a high-priority data source deteriorates, ineffective consumption caused by blindly investing resources is avoided, and instead, dynamic adjustments are made based on its actual accessibility, balancing priority weight and real-time feasibility, and improving the stability and reliability of core data collection.

[0060] Therefore, this embodiment reduces the reliance on manual intervention through a fully automated mechanism that periodically detects, automatically calculates the latest status indicators, and dynamically generates acquisition power. It is especially suitable for acquisition scenarios with multiple data sources and high dynamics, and significantly reduces labor costs.

[0061] Example 2: Compared to Embodiment 1, this embodiment, based on Embodiment 1, further provides a process for determining the priority of the target data source, which includes: G100 calculates the data access popularity of the target data source based on the number of times the data of the target data source is accessed within the target time period; the target time period is a time period ending at the current time and with a preset duration.

[0062] In this embodiment, the preset duration is an empirical value and can be flexibly set according to the business scenario (such as 1 day, 7 days, etc.). For example, it can be set to 1 day for scenarios with high real-time requirements and 7 days for scenarios with low real-time requirements.

[0063] In this embodiment, the target data source refers to the object from which data needs to be collected. The target data source can generate data. Optionally, the target data source is a website.

[0064] In this embodiment, the data access popularity of the target data source refers to the frequency of access to the target data source. The data access popularity of the target data source is positively correlated with the number of times the data of the target data source is accessed within the target time period. As a specific implementation, the number of times the data of the target data source is accessed within the target time period is normalized to obtain the data access popularity of the target data source. Optionally, the data access popularity of the target data source is denoted as H, where H = (cc... min ) / (c max -c min ), where c is the number of times the target data source is accessed within the target time period, c min and c max These are the preset minimum and maximum number of accesses. Therefore, the number of times the target data source is accessed within the target time period can be standardized to the [0,1] interval.

[0065] G200 obtains the data update activity of the target data source based on the update frequency of the target data source within the target time period.

[0066] In this embodiment, the data update activity of the target data source reflects the dynamic characteristics of data updates in the target data source (average level and trend of update frequency), and the dynamic characteristics of data updates in the target data source directly affect data timeliness. As a specific implementation, G200 includes: G210, obtain the update frequency curve of the target data source within the target time period based on the update frequency of the target data source within the target time period; the update frequency of the target data source within the target time period includes the update frequency of the target data source corresponding to each unit of time within the target time period; the horizontal axis of the update frequency curve of the target data source within the target time period is time, and the vertical axis of the update frequency curve of the target data source within the target time period is update frequency.

[0067] As a specific implementation method, the update frequency (i.e., the number of times the data is updated) of the target data source is counted in each unit time within the target time period. An update frequency curve is plotted with time as the horizontal axis and update frequency as the vertical axis. Based on the update frequency curve, the trend of the update frequency of the target data source over time can be obtained.

[0068] G220 retrieves the average absolute value of the slope of the update frequency curve of the target data source within the target time period.

[0069] In this embodiment, the absolute value of the slope of the update frequency curve of the target data source within the target time period can reflect the degree of fluctuation in the update frequency, and the average value of the absolute value of the slope can quantify the overall fluctuation trend. Specifically, the larger the average value of the absolute value of the slope, the more drastic the fluctuation in the update frequency, indicating a stronger ability of the target data source to capture information.

[0070] G230 obtains the data update activity of the target data source based on the average absolute value of the slope of the update frequency curve of the target data source within the target time period and the average update frequency of the target data source within different unit times within the target time period.

[0071] In this embodiment, the update characteristics of the target data source are comprehensively evaluated by combining the average level of update frequency (basic activity) and fluctuation trend (dynamic activity). As a specific implementation, G230 includes: G231 normalizes the average absolute value of the slope of the update frequency curve of the target data source within the target time period to obtain the first activity level of the target data source.

[0072] As a specific implementation method, the first activity level is A1, A1=(zz) min ) / (z max -z min ), where z is the average absolute value of the slope of the update frequency curve of the target data source within the target time period, z min and z max These are the minimum and maximum values ​​of the preset average of the absolute values ​​of the slopes. Therefore, the average of the absolute values ​​of the slopes of the update frequency curve of the target data source within the target time period can be standardized to the [0,1] interval.

[0073] G232 normalizes the average update frequency of the target data source within different unit time periods to obtain the second activity level of the target data source.

[0074] As a specific implementation, the second activity level is A2, A2=(ff min ) / (f max -f min ), where f is the average update frequency of the target data source at different unit times within the target time period, f min and f max These are the minimum and maximum values ​​of the preset update frequency, respectively. Therefore, the average update frequency of the target data source within different unit time periods can be standardized to the [0,1] interval.

[0075] G233 obtains the data update activity of the target data source based on the weight corresponding to the slope of the preset update frequency curve, the first activity of the target data source, the weight corresponding to the average value of the preset update frequency, and the second activity of the target data source.

[0076] As a specific implementation method, the data update activity of the target data source is A, where A = w k ×A1+w f ×A2, w k The weights corresponding to the slope of the preset update frequency curve, w f w is the weight corresponding to the average value of the preset update frequency. k +w f =1, and w k >0, w f >0. w k This reflects the importance of the trend in update frequency fluctuations, w f This reflects the importance of the average update level and can be set empirically, for example, w k =w f =0.5, or w k =0.4, w f =0.6.

[0077] Therefore, this embodiment combines dynamic and static update characteristics to more comprehensively evaluate the data update activity of the target data source.

[0078] G300 is used to obtain the output data volume index of the target data source based on the amount of data generated by the target data source within the target time period.

[0079] In this embodiment, the output data volume index of the target data source reflects the amount of data generated by the target data source within the target time period. The output data volume index of the target data source is positively correlated with the amount of data generated by the target data source within the target time period. The larger the output data volume of the target data source, the richer the information contained in the target data source.

[0080] As a specific implementation method, the total amount of data v (in bytes) generated by the target data source within the target time period is statistically analyzed, and then normalized to obtain the output data volume index v of the target data source. idx , v idx =(vv min ) / (v max -v min ), where v min and v max These are the preset minimum and maximum data volumes, respectively. Therefore, the amount of data collected from the target data source within the target time period can be standardized to the [0,1] interval.

[0081] G400 is used to obtain the data collection resource consumption index of the target data source based on the resources consumed in collecting data from the target data source within the target time period.

[0082] In this embodiment, the data acquisition resource consumption index of the target data source reflects the amount of resources consumed when acquiring data from the target data source. The data acquisition resource consumption index of the target data source is positively correlated with the resources consumed when acquiring data from the target data source within the target time period. The more resources consumed in the acquisition process, the higher the acquisition cost, and resource efficiency needs to be balanced in priority evaluation.

[0083] As a specific implementation method, the resources consumed by the target data source within the target time period are collected for each preset type. Each preset type of resource is then normalized, and the normalization results are weighted and summed to obtain a resource consumption index. Optionally, the preset type of resources includes CPU utilization, memory utilization, and bandwidth utilization. Normalization uses existing normalization methods, such as the min-max normalization method. When weighting and summing the normalization results, the weights corresponding to each preset type of resource are empirical values. The weights corresponding to each preset type of resource are greater than 0, and the sum of the weights corresponding to each preset type of resource is 1. For example, the weight corresponding to each preset type of resource is 1 / R, where R is the number of preset type resources.

[0084] G500 prioritizes target data sources based on their access frequency, update activity, output data volume index, and data acquisition resource consumption index.

[0085] In this embodiment, the priority of the target data source comprehensively evaluates the importance of the target data source. As a specific implementation, G500 includes: G510 obtains the weights corresponding to data access popularity, data update activity, output data volume, and data collection resource consumption; the weights corresponding to data access popularity, data update activity, output data volume, and data collection resource consumption are all greater than 0.

[0086] As a specific implementation method, the weights corresponding to data access popularity, data update activity, data output volume, and data collection resource consumption include: G511, obtain the target variable; the target variable is the manually labeled priority.

[0087] As a specific implementation method, the priority of manually labeled data is collected as the target variable Y (the historical data source is labeled by domain experts according to business needs, and the value range is [0,1]).

[0088] G512 uses four indicators from the target data source as feature variables to construct a dataset containing the correspondence between feature variables and target variables; the four indicators include data access popularity, data update activity, output data volume index, and data collection resource efficiency.

[0089] As a specific implementation method, a dataset is constructed, where each data point includes a feature variable and a corresponding target variable. The data acquisition resource efficiency is defined as the difference between 1 and the data acquisition resource consumption index.

[0090] G513, a linear regression model is trained based on the dataset to obtain the coefficients corresponding to each feature variable in the linear regression model.

[0091] As a specific implementation method, a linear regression model is trained, with the model expression being Y=w h ×x1+w a ×x2+w v ×x3+w r ×x4, w h w represents the feature coefficient corresponding to the data access popularity. a w represents the feature coefficient corresponding to the data update activity. v w represents the characteristic coefficient corresponding to the output data volume index. r Let x1, x2, x3, and x4 be the feature coefficients corresponding to the data acquisition resource consumption index, respectively. x1, x2, x3, and x4 are the feature variables corresponding to data access popularity, data update activity, output data volume index, and data acquisition resource efficiency, respectively. During training, the coefficients for w are added... h 、w a 、w v and w rThe nonnegativity constraint ensures that w h 、w a 、w v and w r The result is positive; the linear regression model is trained using the least squares method. Those skilled in the art will recognize that the specific training process is prior art and will not be described further here.

[0092] G514, normalize the coefficients so that the sum of the normalized coefficients is 1 and each coefficient is greater than 0. Then, determine the normalized coefficients as the weights corresponding to data access popularity, data update activity, output data volume, and data collection resource consumption, respectively.

[0093] As a specific implementation method, the weight corresponding to the data access popularity is w' h w' h =w h / (w h +w a +w v +w r The weight corresponding to the data update activity is w'. a w' a =w a / (w h +w a +w v +w r The weight corresponding to the amount of output data is w'. v w' v =w v / (w h +w a +w v +w r The weight corresponding to the data acquisition resource consumption is w'. r w' r =w r / (w h +w a +w v +w r ).

[0094] Based on G511-G514, this embodiment determines the weights corresponding to data access popularity, data update activity, output data volume, and data collection resource consumption through a data-driven approach. The weights determined in this way are more in line with actual business scenarios.

[0095] G520 determines the first sub-priority of the target data source by multiplying the weight corresponding to the data access popularity by the data access popularity of the target data source.

[0096] G530 determines the second sub-priority of the target data source by multiplying the weight corresponding to the data update activity by the data update activity of the target data source.

[0097] G540 determines the third sub-priority of the target data source by multiplying the weight corresponding to the output data volume by the output data volume index of the target data source.

[0098] G550 determines the fourth sub-priority of the target data source by multiplying the weight corresponding to the data acquisition resource consumption by the data acquisition resource efficiency of the target data source; the data acquisition resource efficiency of the target data source is the difference between 1 and the data acquisition resource consumption index of the target data source.

[0099] G560 determines the priority of the target data source as the sum of the first sub-priority, the second sub-priority, the third sub-priority, and the fourth sub-priority of the target data source.

[0100] Based on G520-G560, the priority of the target data source is obtained by weighted fusion of data access popularity, data update activity, output data volume index and data collection resource efficiency. Thus, the priority of the target data source is determined in a more comprehensive way.

[0101] Example 3: Compared to Embodiment 1, this embodiment, based on Embodiment 1, further provides a process for determining the acquisition power of the target data source based on the acquisition power index of the target data source. This process includes: S510, Obtain the set of experimental crawler programs that have successfully crawled the target data source, which are included in the experimental collection group; the experimental collection group includes several experimental crawler programs.

[0102] In this embodiment, the experimental data collection group is a collection of several experimental web crawler programs. The experimental data collection group includes different experimental web crawler programs and is used to perform preliminary crawling tests on candidate data sources in the candidate data source set, obtaining crawling results to evaluate the data source performance. The candidate data source set contains several data sources to be evaluated (such as websites), and is the target of data collection. The experimental web crawler program is a program used to simulate the data collection process, which can initiate crawling requests to data sources and return success or failure results. The candidate data source is the specific data source to be evaluated, and the target data source is any candidate data source in the candidate data source set. The target data source is the object of evaluation in this embodiment. As a specific implementation, the target data source and candidate data sources are websites.

[0103] In this embodiment, the experimental crawler program and the candidate data source in the experimental data acquisition group meet preset conditions, including compatibility between the experimental crawler program and the candidate data source. This compatibility is a fundamental prerequisite for the effective conduct of the experiment. If the crawler and the data source are incompatible, the crawler will be unable to crawl normally, generating a large amount of invalid and failed data. This not only prevents the selection of a set of successfully crawling experimental crawler programs but also halts the entire experimental process. Based on the preset condition of compatibility between the experimental crawler program and the candidate data source, it ensures that all experimental crawlers can effectively access and crawl the candidate data source, avoiding the inclusion of invalid samples.

[0104] In this embodiment, the candidate data sources in the candidate data source set share the same preset data source attributes. These preset data source attributes include at least one of the following: data source type, anti-crawling strategy, page structure type, access protocol, and request frequency limit. Specifically, the data source type includes static websites, dynamic websites, etc.; the anti-crawling strategy includes CAPTCHA verification, IP restriction, User-Agent detection, etc.; the page structure type includes list structure, detail structure, nested structure, and irregular structure, etc.; the access protocol includes HTTP, FTP, HTTPS, etc.; and the request frequency limit refers to the maximum number of requests a single client is allowed to initiate within a unit of time. This avoids distortion of the experimental crawler's success rate due to differences in data source attributes, ensuring that the success rate accurately reflects the characteristics of the data source itself. This provides an unbiased data sample for subsequently determining the crawling stability of the target data source based on the experimental crawler's success rate.

[0105] In this embodiment, the preset crawler program attributes of different experimental crawler programs in the experimental data collection group are identical. These preset crawler program attributes include at least one of the following: crawling target, running parameters, and validity. The crawling target refers to the specific data content to be obtained from the data source, such as the title, publication time, and body text of a news website. Running parameters refer to the configuration parameters of the crawler program, including concurrency, timeout, and number of retries. Validity refers to the completeness and correctness of the crawler program's functionality. Having identical preset attributes for the experimental crawler programs eliminates the influence of differences between crawlers on the experimental results, ensuring that all experimental crawler programs have consistent crawling capabilities and execution logic. The difference in crawling success rate is determined solely by the true characteristics of the candidate data source, making the subsequent crawling feature judgment index calculated based on the success rate more objective and truly reflecting the stability of the target data source.

[0106] In this embodiment, each experimental crawler program in the experimental acquisition group is used to crawl and test each candidate data source in the candidate data source. This allows us to obtain the crawling results of each experimental crawler program in the experimental acquisition group for each candidate data source. The crawling results include success and failure. The set of experimental crawler programs that successfully crawl the target data source is determined as the experimental crawler program set.

[0107] S520, obtain the success rate of each experimental crawler program in the experimental crawler program set for crawling candidate data sources in the candidate data source set; the candidate data source set includes several candidate data sources.

[0108] In this embodiment, the crawling results of each experimental crawler program in the experimental crawler program set for each candidate data source in the candidate data source set are known. Based on this, the crawling success rate of each experimental crawler program in the experimental crawler program set for each candidate data source in the candidate data source set can also be obtained. The crawling success rate of any experimental crawler program in the experimental crawler program set for each candidate data source in the candidate data source set is the ratio of the number of successful crawls of that experimental crawler program for each candidate data source in the candidate data source set to the number of candidate data sources included in the candidate data source set.

[0109] S530, determine the crawling feature judgment index of the target data source based on the crawling success rate.

[0110] As one specific implementation, S530 includes: S531, obtain the average success rate of the experimental crawler programs in the experimental crawler program set for crawling the candidate data sources in the candidate data source set.

[0111] S532, obtain the variance of the success rate of the experimental crawler programs in the experimental crawler program set to crawl the candidate data sources in the candidate data source set.

[0112] S533, determine the crawling feature judgment index of the target data source based on the average and variance of the crawling success rate; the crawling feature judgment index of the target data source is negatively correlated with the average of the crawling success rate; the crawling feature judgment index of the target data source is negatively correlated with the variance of the crawling success rate.

[0113] As a specific implementation method, the average and variance of the crawling success rate are normalized respectively. Then, the difference between 1 and the normalized result of the average is determined as the first exponent, and the difference between 1 and the normalized result of the variance is determined as the second exponent. The product of the first exponent and the second exponent is determined as the crawling feature judgment index of the target data source. Those skilled in the art will know that the normalization process is prior art and will not be described in detail here.

[0114] In this embodiment, the larger the crawling feature judgment index of the target data source, the easier the target data source is to be crawled and the more stable the crawling difficulty is; conversely, the smaller the index, the less likely the target data source is to be crawled or the less stable the crawling difficulty is.

[0115] S540, if the crawling feature judgment index of the target data source is greater than or equal to a preset judgment index threshold, then the collection power of the target data source is determined based on the comparison result of the collection power index of the target data source and the preset collection power index threshold; otherwise, the collection power of the target data source is determined based on the comparison result of the collection power index of the target data source and the updated collection power index threshold; the updated collection power index threshold is obtained based on the preset collection power index threshold, and the updated collection power index threshold is not equal to the preset collection power index threshold.

[0116] As a specific implementation method, the acquisition power index of the target data source is a comprehensive indicator that quantitatively describes the intensity of resource investment and the aggressiveness of the strategy when acquiring data from the target data source. The range of the acquisition power index of the target data source is [0,1], which can be set manually.

[0117] As a specific implementation, determining the acquisition power of the target data source based on the comparison result of the acquisition power index of the target data source and a preset acquisition power index threshold includes: if the acquisition power index of the target data source is greater than or equal to a preset first acquisition power index threshold, then the acquisition power of the target data source is determined to be a preset maximum acquisition power; if the acquisition power index of the target data source is less than or equal to a preset second acquisition power index threshold, then the acquisition power of the target data source is determined to be a preset minimum acquisition power; wherein the preset first acquisition power index threshold is greater than the preset second acquisition power index threshold.

[0118] As a specific implementation, determining the acquisition power of the target data source based on the comparison result of the acquisition power index of the target data source and a preset acquisition power index threshold further includes: if the acquisition power index of the target data source is less than a preset first acquisition power index threshold and greater than a preset second acquisition power index threshold, then the acquisition power index of the target data source is normalized according to the preset first acquisition power index threshold and the preset second acquisition power index threshold, and the product of the normalized acquisition power index of the target data source and a preset maximum acquisition power is determined as the acquisition power of the target data source. It should be understood that the normalization process here refers to maximum-minimum value normalization.

[0119] In one specific implementation, determining the acquisition power of the target data source based on a comparison between the acquisition power index of the target data source and an updated acquisition power index threshold includes: if the acquisition power index of the target data source is greater than or equal to the updated first acquisition power index threshold, then the acquisition power of the target data source is determined to be a preset maximum acquisition power; if the acquisition power index of the target data source is less than or equal to the updated second acquisition power index threshold, then the acquisition power of the target data source is determined to be a preset minimum acquisition power; wherein the updated first acquisition power index threshold is greater than the updated second acquisition power index threshold; the updated first acquisition power index threshold is less than the preset first acquisition power index threshold, and the updated second acquisition power index threshold is less than the preset second acquisition power index threshold.

[0120] As a specific implementation, determining the acquisition power of the target data source based on the comparison result of the acquisition power index of the target data source and the updated acquisition power index threshold further includes: if the acquisition power index of the target data source is less than the updated first acquisition power index threshold and greater than the updated second acquisition power index threshold, then the acquisition power index of the target data source is normalized according to the updated first acquisition power index threshold and the updated second acquisition power index threshold, and the larger value of the product of the normalized acquisition power index of the target data source and the preset maximum acquisition power and the original acquisition power is determined as the acquisition power of the target data source; wherein, the original acquisition power is: based on the normalization of the acquisition power index of the target data source according to the preset first acquisition power index threshold and the preset second acquisition power index threshold, and the product of the normalized acquisition power index of the target data source and the preset maximum acquisition power. It should be understood that the normalization process here refers to the maximum value minus the minimum value normalization process.

[0121] As a specific implementation, T’1 is the updated first acquisition power index threshold, where 0 < T’1 < 1, and T’1 = (1 - k1×(k - k0) / k)×T1. Here, T1 is the preset first acquisition power index threshold, 0 < T1 < 1, k1 is the preset first adjustment coefficient, 0 < k1 < 1, k is the preset determination index threshold, k0 is the crawl feature determination index of the target data source, and k0 < k. Optionally, T1, k1, and k are empirical values. For example, T1 = 0.8, k1 = 0.5, and k = 0.6. Based on this formula, when the actual crawl feature determination index of the target data source is less than the preset determination index threshold, the updated first acquisition power index threshold is less than the preset first acquisition power index threshold; and the more the actual crawl feature determination index of the target data source is less than the preset determination index threshold, the more the updated first acquisition power index threshold is less than the preset first acquisition power index threshold. This helps to enable the strong mode earlier to cope with possible crawling difficulties when the data source is unstable or difficult to crawl.

[0122] As a specific implementation, T’2 is the updated second acquisition power index threshold, where 0 < T’2 < 1, and T’2 = (1 - k2×(k - k0) / k)×T2. Here, T2 is the preset second acquisition power index threshold, 0 < T2 < 1, k2 is the preset second adjustment coefficient, 0 < k2 < 1, k is the preset determination index threshold, k0 is the crawl feature determination index of the target data source, k0 < k, and (1 - k1)×T1 > (1 - k2)×T2. Optionally, T2 and k2 are empirical values. For example, T2 = 0.3 and k2 = 0.3. Based on this formula, the minimum power acquisition instruction is triggered only when the acquisition power index drops to a lower level. This forces the system to adopt a strategy of increasing the sampling power when the data source is unstable or difficult to crawl, making it less likely to enter the low-power or sleep state and giving priority to ensuring successful crawling.

[0123] This embodiment obtains the crawling success rate of each experimental crawler program in the experimental crawler program set for the candidate data sources in the candidate data source set. Based on the crawling success rate, a crawling feature judgment index for the target data source is determined. If the crawling feature judgment index of the target data source is greater than or equal to a preset judgment index threshold, the acquisition power of the target data source is determined by comparing the acquisition power index of the target data source with the preset acquisition power index threshold; otherwise, the acquisition power of the target data source is determined by comparing the acquisition power index of the target data source with the updated acquisition power index threshold. Therefore, this embodiment, by introducing a crawling feature judgment index and constructing a dynamic threshold adjustment mechanism, can intelligently distinguish the stability characteristics of data sources. For data sources with stable characteristics, a conventional threshold is used for decision-making; for data sources with unstable characteristics, an updated threshold is used. This differentiated processing method helps improve the accuracy of power decision-making, thereby improving the overall success rate and efficiency of data acquisition.

[0124] Example 4: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: Obtain the experimental data collection group; the experimental data collection group includes several experimental crawler programs.

[0125] According to the preset detection cycle, the experimental acquisition group periodically probes the target data source, and obtains the detection result corresponding to each detection based on the crawling result of each experimental crawler program on the target data source for each detection; the detection result corresponding to each detection includes the number of successful crawls and the crawling success rate; the crawling result of any experimental crawler program on the target data source for each detection is either successful or unsuccessful.

[0126] Based on the detection results of the most recent preset detection, obtain the latest number of successful crawls and the latest crawl success rate of the target data source.

[0127] The collection power index of the target data source is obtained based on the latest number of successful crawls, the latest success rate of crawling, and the priority of the target data source.

[0128] The acquisition power of the target data source is determined based on the acquisition power index of the target data source, and data is acquired from the target data source based on the acquisition power of the target data source.

[0129] Example 5: This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: Obtain the experimental data collection group; the experimental data collection group includes several experimental crawler programs.

[0130] According to the preset detection cycle, the experimental acquisition group periodically probes the target data source, and obtains the detection result corresponding to each detection based on the crawling result of each experimental crawler program on the target data source for each detection; the detection result corresponding to each detection includes the number of successful crawls and the crawling success rate; the crawling result of any experimental crawler program on the target data source for each detection is either successful or unsuccessful.

[0131] Based on the detection results of the most recent preset detection, obtain the latest number of successful crawls and the latest crawl success rate of the target data source.

[0132] The collection power index of the target data source is obtained based on the latest number of successful crawls, the latest success rate of crawling, and the priority of the target data source.

[0133] The acquisition power of the target data source is determined based on the acquisition power index of the target data source, and data is acquired from the target data source based on the acquisition power of the target data source.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0135] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A data acquisition and processing method, characterized in that, The method includes the following steps: Obtain the experimental data collection group; the experimental data collection group includes several experimental web crawler programs; According to the preset detection cycle, the experimental acquisition group periodically detects the target data source, and obtains the detection result corresponding to each detection based on the crawling result of each experimental crawler program on the target data source for each detection; the detection result corresponding to each detection includes the number of successful crawls and the crawling success rate; the crawling result of any experimental crawler program on the target data source for each detection is either successful or unsuccessful. Based on the detection results corresponding to the most recent preset detection, obtain the latest number of successful crawls and the latest success rate of crawling the target data source; The collection power index of the target data source is obtained based on the latest number of successful crawls, the latest crawl success rate, and the priority of the target data source. The acquisition power of the target data source is determined based on the acquisition power index of the target data source, and data is acquired from the target data source based on the acquisition power of the target data source.

2. The data acquisition and processing method according to claim 1, characterized in that, The detection results corresponding to each detection are obtained based on the crawling results of each experimental crawler program on the target data source for each detection, including: For any given probe, the number of experimental crawler programs in the experimental collection group corresponding to that probe that successfully crawled the target data source is defined as the number of successful crawls of the target data source; the ratio of the number of successful crawls of the target data source to the number of experimental crawler programs included in the experimental collection group is defined as the success rate of crawling the target data source.

3. The data acquisition and processing method according to claim 1, characterized in that, The number of successful crawls of the target data source is positively correlated with the data collection power index of the target data source, the crawl success rate of the target data source is negatively correlated with the data collection power index of the target data source, and the priority of the target data source is positively correlated with the data collection power index of the target data source.

4. The data acquisition and processing method according to claim 3, characterized in that, The collection power index of the target data source is obtained based on the latest successful crawl count, latest success rate, and priority. The ratio of the first dynamic coefficient to the latest crawling success rate of the target data source is determined as the first collection power sub-index of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1; The ratio of the latest successful crawl count of the target data source to the second dynamic coefficient is determined as the second acquisition power sub-index of the target data source; the second acquisition power sub-index is greater than or equal to 0 and less than or equal to 1; The acquisition power index of the target data source is obtained based on the first acquisition power sub-index, the second acquisition power sub-index, and the priority of the target data source; the acquisition power index of the target data source is positively correlated with the first acquisition power sub-index, and the acquisition power index of the target data source is positively correlated with the second acquisition power sub-index.

5. The data acquisition and processing method according to claim 4, characterized in that, The first dynamic coefficient is positively correlated with the latest crawling success rate of the target data source.

6. The data acquisition and processing method according to claim 4, characterized in that, The second dynamic coefficient is determined based on the maximum number of successful crawls of the target data source during the target historical time period.

7. The data acquisition and processing method according to claim 4, characterized in that, The acquisition power index of the target data source is determined by the product of the first acquisition power sub-index, the second acquisition power sub-index, and the priority of the target data source.

8. The data acquisition and processing method according to claim 1, characterized in that, The acquisition power index of the target data source is positively correlated with the acquisition power of the target data source.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data acquisition processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data acquisition processing method as described in any one of claims 1 to 8.

Images