A data source collection method and device based on data source priority and a medium

Abstract

The application relates to the technical field of electric digital data processing, in particular to a data source collection method based on data source priority, a device and a medium. The method comprises the following steps: acquiring data access heat of a target data source according to the number of times that data of the target data source is accessed in a target time period; acquiring data update activity of the target data source according to the update frequency of the target data source in the target time period; acquiring output data amount index of the target data source according to the data amount generated by collecting the target data source in the target time period; acquiring data collection resource consumption index of the target data source according to the resources consumed by collecting data of the target data source in the target time period; acquiring priority of the target data source according to the data access heat, the data update activity, the output data amount index and the data collection resource consumption index of the target data source; and collecting data of the target data source according to the priority of the target data source. The application can improve the utilization rate of collection resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, and in particular to a data source collection method based on data source priority, a device and a medium. BACKGROUND

[0002] In today's era of big data and artificial intelligence, enterprises and organizations increasingly rely on collecting, processing and analyzing data from a large number of data sources to drive business decisions, optimize operations and develop intelligent applications. Different data sources have different characteristics, for example, some data sources have slow update frequency, while some data sources have fast update frequency; some data sources generate large amounts of data, while some data sources generate small amounts of data. If the same collection method is developed for data sources with different characteristics, such as allocating the same collection resources, some data sources may waste collection resources, while some data sources may lack collection resources, resulting in low utilization of collection resources. The Chinese patent application with application number 202010249368.7 discloses a business data updating method, device, equipment and storage medium, which flexibly sets the update frequency of different data sources of different enterprises, avoids wasting update resources by updating all data uniformly, and improves data update efficiency. The patent application determines the update frequency based on the importance of data, the acquisition cost and the risk level of the data source, which is relatively limited to the financial field and is not suitable for scenarios where the data source is a website. How to provide another collection method that can improve the utilization rate of collection resources is a problem to be solved. SUMMARY

[0003] The present application aims to provide a data source collection method based on data source priority, a device and a medium to improve the utilization rate of collection resources.

[0004] According to a first aspect of the present application, a data source collection method based on data source priority is provided, the method comprising the following steps:

[0005] According to the number of times the data of the target data source is accessed within the target time period, the data access popularity of the target data source is obtained; the target time period is a time period with a preset length ending at the current time.

[0006] According to the update frequency of the target data source within the target time period, the data update activity of the target data source is obtained.

[0007] According to the amount of data generated by collecting the target data source within the target time period, the output data amount index of the target data source is obtained.

[0008] According to the resources consumed by collecting the data of the target data source within the target time period, the data collection resource consumption index of the target data source is obtained.

[0009] According to the data access heat, data update activity, output data volume index and data collection resource consumption index of the target data source, a priority of the target data source is obtained.

[0010] According to the priority of the target data source, data collection is performed on the target data source.

[0011] Further, the data update activity of the target data source according to the update frequency of the target data source in the target time period comprises:

[0012] An update frequency curve of the target data source in the target time period is obtained according to the update frequency of the target data source in the target time period; the update frequency of the target data source in the target time period comprises the update frequency of the target data source corresponding to each unit time in the target time period; the horizontal axis of the update frequency curve of the target data source in the target time period is time, and the vertical axis of the update frequency curve of the target data source in the target time period is update frequency.

[0013] An average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period is obtained.

[0014] The data update activity of the target data source is obtained according to the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period and the average value of the update frequency of the target data source in different unit times in the target time period.

[0015] Further, the data update activity of the target data source according to the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period and the average value of the update frequency of the target data source in different unit times in the target time period comprises:

[0016] The average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period is normalized to obtain a first activity of the target data source.

[0017] The average value of the update frequency of the target data source in different unit times in the target time period is normalized to obtain a second activity of the target data source.

[0018] The data update activity of the target data source is obtained according to the preset weight corresponding to the slope of the update frequency curve, the first activity of the target data source, the preset weight corresponding to the average value of the update frequency, and the second activity of the target data source.

[0019] Further, the data access heat of the target data source is positively correlated with the number of times that the data of the target data source is accessed in the target time period; the output data volume index of the target data source is positively correlated with the data volume generated by the target data source in the target time period; and the data collection resource consumption index of the target data source is positively correlated with the resources consumed by the data of the target data source in the target time period.

[0020] Further, the priority of the target data source is obtained according to the data access heat of the target data source, the data update activity of the target data source, the output data volume index of the target data source, and the data collection resource consumption index of the target data source, and the priority of the target data source comprises:

[0021] obtaining a weight corresponding to the data access heat, a weight corresponding to the data update activity, a weight corresponding to the output data volume, and a weight corresponding to the data collection resource consumption; the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption are all greater than 0.

[0022] determining the product of the weight corresponding to the data access heat and the data access heat of the target data source as a first sub-priority of the target data source.

[0023] determining the product of the weight corresponding to the data update activity and the data update activity of the target data source as a second sub-priority of the target data source.

[0024] determining the product of the weight corresponding to the output data volume and the output data volume index of the target data source as a third sub-priority of the target data source.

[0025] determining the product of the weight corresponding to the data collection resource consumption and the data collection resource efficiency of the target data source as a fourth sub-priority of the target data source; the data collection resource efficiency of the target data source is the difference between 1 and the data collection resource consumption index of the target data source.

[0026] determining the sum of the first sub-priority of the target data source, the second sub-priority of the target data source, the third sub-priority of the target data source, and the fourth sub-priority of the target data source as the priority of the target data source.

[0027] Further, obtaining the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption comprises:

[0028] obtaining a target variable; the target variable is a manually labeled priority.

[0029] using four indexes of the target data source as characteristic variables to construct a data set containing a corresponding relationship between the characteristic variables and the target variable; the four indexes include the data access heat, the data update activity, the output data volume index, and the data collection resource efficiency.

[0030] training a linear regression model based on the data set to obtain coefficients corresponding to each characteristic variable in the linear regression model;

[0031] The coefficients are normalized so that the sum of the normalized coefficients is 1 and each coefficient is greater than 0, and the normalized coefficients are respectively determined as a weight corresponding to data access heat, a weight corresponding to data update activity, a weight corresponding to output data volume, and a weight corresponding to data collection resource consumption.

[0032] Further, the linear regression model is trained by a least square method.

[0033] Further, the data collection on the target data source according to the priority of the target data source comprises: acquiring a collection power of the target data source according to the priority of the target data source, and collecting data of the target data source based on the collection power of the target data source; the collection power of the target data source is positively correlated with the priority of the target data source.

[0034] According to a second aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data source collection method based on the priority of the data source when executing the computer program.

[0035] According to a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executable on a processor to implement the data source collection method based on the priority of the data source.

[0036] Compared with the prior art, the present application has at least the following beneficial effects:

[0037] The present application acquires the priority of the target data source according to the data access heat, the data update activity, the output data volume index and the data collection resource consumption index of the target data source, and collects the target data source according to the priority of the target data source; thus, when collecting the target data source, the present application considers four dimensional characteristics of the data access heat (reflecting the demand degree of data), the data update activity (reflecting the timeliness demand of data), the output data volume index (reflecting the value of data scale) and the data collection resource consumption index (reflecting the collection cost), which is beneficial to improve the matching degree of the collection resource allocated to the target data source and the target data source, and improve the utilization rate of the collection resource. Moreover, the present application considers the above characteristics of the target data source in the latest historical time period (i.e. the target time period), which can realize the dynamic adjustment of the collection resource of the target data source, so that the collection resource allocated to the target data source matches the latest characteristics of the target data source, and further improves the utilization rate of the collection resource. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0039] Figure 1 The flow chart of the data source collection method based on data source priority provided for the first embodiment of the present application;

[0040] Figure 2 The flow chart of the step of obtaining the data update activity of the target data source provided for the first embodiment of the present application;

[0041] Figure 3 The flow chart of the step of obtaining the priority of the target data source provided for the first embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0043] Embodiment one:

[0044] According to the present embodiment, as shown in Figure 1 , a data source collection method based on data source priority is provided, which comprises the following steps:

[0045] S100, obtaining the data access heat of the target data source according to the number of times that the data of the target data source is accessed within a target time period; the target time period is a time period with the current time as the end point and a preset time length.

[0046] In the present embodiment, the preset time length is an empirical value, which can be flexibly set according to the business scenario (such as 1 day, 7 days, etc.). For example, for a scenario with high real-time requirement, 1 day can be set, and for a scenario with low real-time requirement, 7 days can be set.

[0047] In the present embodiment, the target data source refers to an object that needs to be collected, and the target data source can generate data. Optionally, the target data source is a website.

[0048] In this embodiment, the data access heat of the target data source refers to the frequency of access to the target data source, and the data access heat of the target data source is positively correlated with the number of times of access to the data of the target data source in the target time period. As a specific implementation, the number of times of access to the data of the target data source in the target time period is normalized to obtain the data access heat of the target data source. Optionally, the data access heat of the target data source is H, H = (c - c min ) / (c max -c min ), where c is the number of times of access to the data of the target data source in the target time period, c min and c max are the preset minimum access number and maximum access number, respectively. Thus, the number of times of access to the data of the target data source in the target time period can be standardized to the interval [0, 1].

[0049] S200, obtaining the data update activity of the target data source according to the update frequency of the target data source in the target time period.

[0050] In this embodiment, the data update activity of the target data source reflects the dynamic characteristics (average level and change trend) of the data update of the target data source, and the dynamic characteristics of the data update of the target data source directly affect the data timeliness. As a specific implementation, as shown in Figure 2 S200 includes:

[0051] S210, obtaining the update frequency curve of the target data source in the target time period according to the update frequency of the target data source in the target time period; the update frequency of the target data source in the target time period includes the update frequency of the target data source corresponding to each unit time in the target time period; the horizontal axis of the update frequency curve of the target data source in the target time period is time, and the vertical axis of the update frequency curve of the target data source in the target time period is update frequency.

[0052] As a specific implementation, the update frequency (i.e., the number of data updates) of the target data source in each unit time included in the target time period is counted, and the update frequency curve is drawn with time as the horizontal axis and update frequency as the vertical axis; based on the update frequency curve, the change trend of the update frequency of the target data source over time can be obtained.

[0053] S220, obtaining the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period.

[0054] In this embodiment, the absolute value of the slope of the update frequency curve of the target data source in the target time period can reflect the fluctuation degree of the update frequency, and the average value of the absolute value of the slope can quantify the overall fluctuation trend. The greater the average value of the absolute value of the slope, the more intense the update frequency fluctuation, and the stronger the information capturing ability of the target data source.

[0055] In S230, the data update activity of the target data source is obtained according to the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period and the average value of the update frequency of the target data source in different unit time in the target time period.

[0056] In this embodiment, the average level of the update frequency (basic activity) and the fluctuation trend (dynamic activity) are combined to comprehensively evaluate the update characteristics of the target data source. As a specific implementation, S230 includes:

[0057] In S231, the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period is normalized to obtain the first activity of the target data source.

[0058] As a specific implementation, the first activity is A1, A1=(z-z min ) / (z max -z min ), where z is the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period, z min and z max are the minimum and maximum values of the average value of the absolute value of the slope preset respectively. Thus, the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period can be standardized to the interval [0, 1].

[0059] In S232, the average value of the update frequency of the target data source in different unit time in the target time period is normalized to obtain the second activity of the target data source.

[0060] As a specific implementation, the second activity is A2, A2=(f-f min ) / (f max -f min ), where f is the average value of the update frequency of the target data source in different unit time in the target time period, f min and f max are the minimum and maximum values of the average value of the update frequency preset respectively. Thus, the average value of the update frequency of the target data source in different unit time in the target time period can be standardized to the interval [0, 1].

[0061] S233, acquiring the data update activity of the target data source according to the weight corresponding to the slope of the preset update frequency curve, the first activity of the target data source, the weight corresponding to the average value of the preset update frequency, and the second activity of the target data source.

[0062] As a specific embodiment, the data update activity of the target data source is A, A = w k ×A1+w f ×A2, w k is the weight corresponding to the slope of the preset update frequency curve, w f is the weight corresponding to the average value of the preset update frequency, w k +w f =1, and w k >0, w f >0. w k reflects the importance of the update frequency fluctuation trend, w f reflects the importance of the average update level, which can be set according to experience, for example, w k =w f =0.5, or w k =0.4, w f =0.6.

[0063] Thus, the embodiment comprehensively evaluates the data update activity of the target data source by combining dynamic and static update characteristics.

[0064] S300, acquiring the output data amount index of the target data source according to the amount of data generated by the target data source in the target time period.

[0065] In the embodiment, the output data amount index of the target data source reflects the amount of data generated by the target data source in the target time period, and the output data amount index of the target data source is positively correlated with the amount of data generated by the target data source in the target time period. The greater the output data amount of the target data source, the richer the information contained in the target data source.

[0066] As a specific embodiment, the total data amount v generated by the target data source in the target time period is counted (for example, in bytes), and the output data amount index v idx of the target data source is obtained by normalization processing, v idx =(v-v min ) / (v max -v min ), where v min and v max are the preset minimum data amount and maximum data amount, respectively. Thus, the amount of data generated by the target data source in the target time period can be standardized to the interval [0, 1].

[0067] S400, acquiring a data collection resource consumption index of the target data source according to resources consumed in collecting data of the target data source in the target time period.

[0068] In this embodiment, the data collection resource consumption index of the target data source reflects how much resources are consumed in collecting data of the target data source, and the data collection resource consumption index of the target data source is positively correlated with resources consumed in collecting data of the target data source in the target time period. The more resources consumed in the collection process, the higher the collection cost, which needs to be balanced in the priority evaluation.

[0069] As a specific implementation, each preset type of resources consumed in collecting the target data source in the target time period is counted, each preset type of resources is normalized, and the normalized results are weighted and summed to obtain the resource consumption index. Optionally, the preset type of resources includes CPU occupancy, memory occupancy, and bandwidth occupancy, the normalization uses an existing normalization method, for example, a min-max normalization method; the weight of each preset type of resource used when the normalized results are weighted and summed is an empirical value, the weight of each preset type of resource is greater than 0, the sum of the weights of each preset type of resource is 1, for example, the weight of each preset type of resource is 1 / R, and R is the number of preset types of resources.

[0070] S500, acquiring the priority of the target data source according to the data access heat, the data update activity, the output data volume index, and the data collection resource consumption index of the target data source.

[0071] In this embodiment, the priority of the target data source comprehensively evaluates the importance of the target data source, and as a specific implementation, as shown in Figure 3 S500 includes:

[0072] S510, acquiring a weight corresponding to the data access heat, a weight corresponding to the data update activity, a weight corresponding to the output data volume, and a weight corresponding to the data collection resource consumption; the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption are all greater than 0.

[0073] As a specific implementation, acquiring the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption includes:

[0074] S511, acquiring a target variable; the target variable is a manually labeled priority.

[0075] As a specific embodiment, the priority of the artificial labeling is collected as the target variable Y (labeled by a domain expert according to business requirements on a historical data source, with a value range of [0, 1]).

[0076] S512, four indicators of the target data source are taken as feature variables to construct a data set containing the corresponding relationship between the feature variables and the target variable; the four indicators include data access heat, data update activity, output data volume index, and data collection resource efficiency.

[0077] As a specific embodiment, the data set is constructed, and each piece of data includes a feature variable and a corresponding target variable. The data collection resource efficiency is the difference between 1 and the data collection resource consumption index.

[0078] S513, a linear regression model is trained based on the data set to obtain the coefficients of each feature variable in the linear regression model.

[0079] As a specific embodiment, the linear regression model is trained, and the model expression is Y = w h x1 + w a x2 + w v x3 + w r x4, w h is the feature coefficient corresponding to the data access heat, w a is the feature coefficient corresponding to the data update activity, w v is the feature coefficient corresponding to the output data volume index, and w r is the feature coefficient corresponding to the data collection resource consumption index. x1, x2, x3, and x4 are respectively the feature variable corresponding to the data access heat, the feature variable corresponding to the data update activity, the feature variable corresponding to the output data volume index, and the feature variable corresponding to the data collection resource efficiency. In the training process, non-negative constraints are added to w h , w a , w v , and w r to ensure that w h , w a , w v , and w r are positive; the linear regression model is trained by the least square method. Those skilled in the art know that the specific training process is prior art, which will not be described here.

[0080] S514, the coefficients are normalized so that the sum of the normalized coefficients is 1 and each coefficient is greater than 0, and the normalized coefficients are respectively determined as the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption.

[0081] As a specific embodiment, the weight corresponding to the data access heat is w h , the weight corresponding to the data update activity is w h , the weight corresponding to the output data volume is w h , and the weight corresponding to the data collection resource consumption is w h . a v r a a a h a v r v v v h a v r r r r h a v r

[0082] Based on S511-S514, the embodiment determines the weight corresponding to the data access heat, the weight corresponding to the data update activity, the weight corresponding to the output data volume, and the weight corresponding to the data collection resource consumption in a data-driven manner, so that the determined weights can be more consistent with actual business scenarios.

[0083] S520, the product of the weight corresponding to the data access heat and the data access heat of the target data source is determined as the first sub-priority of the target data source.

[0084] S530, the product of the weight corresponding to the data update activity and the data update activity of the target data source is determined as the second sub-priority of the target data source.

[0085] S540, the product of the weight corresponding to the output data volume and the output data volume index of the target data source is determined as the third sub-priority of the target data source.

[0086] S550, the product of the weight corresponding to the data collection resource consumption and the data collection resource efficiency of the target data source is determined as the fourth sub-priority of the target data source; the data collection resource efficiency of the target data source is the difference between 1 and the data collection resource consumption index of the target data source.

[0087] ​​​​​​​​​​​​​​​​​​​​​​​​S560, determine the sum of the first sub-priority of the target data source, the second sub-priority of the target data source, the third sub-priority of the target data source and the fourth sub-priority of the target data source as the priority of the target data source.

[0088] Based on S520-S560, the priority of the target data source is obtained by weightedly fusing the data access heat, the data update activity, the output data volume index and the data collection resource efficiency of the target data source, so that the priority of the target data source determined considers more comprehensive dimensions.

[0089] S600, collecting data of the target data source according to the priority of the target data source.

[0090] As a specific embodiment, collecting data of the target data source according to the priority of the target data source includes: obtaining the collection power of the target data source according to the priority of the target data source, and collecting data of the target data source based on the collection power of the target data source; the collection power of the target data source is positively correlated with the priority of the target data source.

[0091] In this embodiment, the collection power of the target data source reflects the collection intensity of the target data source, and the collection power of the target data source can be a collection frequency; the collection power of the target data source is positively correlated with the priority of the target data source, so that the data source with higher priority can obtain more collection resources, and the data of the data source with higher priority can be collected preferentially.

[0092] As a specific embodiment, a mapping relationship between the priority and the collection power is set in advance, for example, when the priority of the target data source is greater than or equal to a first preset priority threshold, a first preset collection power is determined as the collection power of the target data source; when the priority of the target data source is less than the first preset priority threshold and greater than a second preset priority threshold, a second preset collection power is determined as the collection power of the target data source; when the priority of the target data source is less than or equal to the second preset priority threshold, a third preset collection power is determined as the collection power of the target data source; wherein the first preset priority threshold is greater than the second preset priority threshold, the first preset collection power is greater than the second preset collection power, and the second preset collection power is greater than the third preset collection power; optionally, the first preset priority threshold, the second preset priority threshold, the first preset collection power, the second preset collection power and the third preset collection power are all empirical values, for example, the first preset priority threshold is 0.8, the second preset priority threshold is 0.5, the collection frequency corresponding to the first preset collection power is 1 / 10 minutes, the second preset collection power is 1 / 30 minutes, and the third preset collection power is 1 / 60 minutes.

[0093] Therefore, the embodiment can realize the optimized allocation of resources, and preferentially guarantee the collection of high importance data sources under limited resources, and improve the overall value of data collection.

[0094] The application obtains the priority of the target data source according to the data access heat, data update activity, output data volume index and data collection resource consumption index of the target data source, and collects the target data source according to the priority of the target data source.

[0095] Embodiment two:

[0096] Compared with embodiment two, the embodiment further provides a process of obtaining the collection power of the target data source according to the priority of the target data source on the basis of embodiment one, and the process comprises:

[0097] S610, obtaining an experimental collection group; the experimental collection group comprises a plurality of experimental crawler programs.

[0098] In the embodiment, the experimental collection group is a set composed of a plurality of experimental crawler programs, and the experimental collection group comprises different experimental crawler programs. The experimental collection group is used for preliminary crawling test on the target data source, and obtains a crawling result to evaluate the performance of the target data source. The experimental crawler program is a program used for simulating a data collection process, and can initiate a crawling request to the target data source and return a success or failure result.

[0099] The target data source is a data source (such as a website) to be evaluated, and is a target of data collection.

[0100] The embodiment can improve the accuracy of evaluation on the target data source by performing crawling test on the target data source based on different experimental crawler programs.

[0101] S620, periodically probe the target data source according to a preset probing period through the experimental collection group, and obtain the probing result corresponding to each probing according to the crawling result of each experimental crawler program corresponding to the probing on the target data source; the probing result corresponding to each probing includes the number of successful crawlings and the crawling success rate; the crawling result of any experimental crawler program corresponding to each probing on the target data source is success or failure.

[0102] In this embodiment, the probing period refers to the time interval between two adjacent probings, and the probing period is used to control the probing frequency on the target data source. The preset probing period is a pre-set time parameter; optionally, the preset probing period is an empirical value, for example, 10 minutes / time or 30 minutes / time. In each probing period, the experimental crawler program is controlled to initiate a standardized crawling request to the target data source, and the crawling result (success / failure) is recorded.

[0103] In this embodiment, the periodic probing refers to the process of periodically initiating a standardized crawling request to the target data source by the experimental collection group according to the preset probing period. For each probing, the number of experimental crawler programs in the experimental collection group corresponding to the probing that have successful crawling results on the target data source is determined as the number of successful crawlings of the target data source; and the ratio of the number of successful crawlings of the target data source to the number of experimental crawler programs included in the experimental collection group is determined as the crawling success rate of the target data source.

[0104] In this embodiment, the carrying capacity and the crawlable performance of the target data source are evaluated based on the number of successful crawlings and the success rate of the target data source, and the carrying capacity and the crawlable performance of the target data source are converted into quantifiable indicators, which provides data basis for subsequent determination of the collection power index of the target data source.

[0105] S630, obtain the latest crawling success number and the latest crawling success rate of the target data source according to the probing result corresponding to the latest preset probing.

[0106] In this embodiment, the preset number refers to the number of the latest probing results referred to when calculating the latest crawling success number and the latest crawling success rate, which is a pre-set number parameter. The preset number is used to filter historical obsolete data, focus on the recent state of the data source, and balance timeliness and stability; optionally, the preset number is an empirical value, for example, the preset number is 1 or 3 or 5.

[0107] As a specific implementation, the latest crawling success number is B, B = ∑ N j=1 (δ j ×B j ), B j is the crawling success number corresponding to the latest jth probing, δ jThe weight corresponding to the latest jth detection, j ranges from 1 to N, N is a preset number of times; optionally, δ j =1 / N. The latest crawling success rate is p, p=∑ N j=1 (δ j ×p j ), p j is the crawling success rate corresponding to the latest jth detection.

[0108] S640, according to the latest crawling success number, the latest crawling success rate and the priority of the target data source, obtaining the collection power index of the target data source.

[0109] In this embodiment, the collection power index of the target data source is an index for comprehensively measuring the priority, the latest crawling success number and the latest crawling success rate of the target data source. For the target data source, the crawling success number of the target data source is positively correlated with the collection power index of the target data source, the crawling success rate of the target data source is negatively correlated with the collection power index of the target data source, and the priority of the target data source is positively correlated with the collection power index of the target data source.

[0110] As a specific implementation, S640 includes:

[0111] S641, determining the ratio of the first dynamic coefficient and the latest crawling success rate of the target data source as the first collection power sub-index of the target data source; the first collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0112] In this embodiment, the first dynamic coefficient is a dynamic adjustment parameter related to the latest crawling success rate of the target data source, and as a specific implementation, the first dynamic coefficient is positively correlated with the latest crawling success rate of the target data source.

[0113] In this embodiment, the first collection power sub-index reflects the contribution of the success rate to the collection power index, and the first collection power sub-index is I1, I1=α / p, the value range of I1 is [0, 1].

[0114] As a specific implementation, the first dynamic coefficient is α, α=(e b×(p-1) -e -b ) / (1-e -b), p is a latest successful crawling rate of the target data source, b is a preset parameter value, and b > 0. Optionally, b is an experience value, for example, b = 2. Based on the formula, a increases with the increase of the latest successful rate p of the target data source, when p → 0 (the latest successful rate of the target data source is extremely low), a ≈ 0, so I1 ≈ 0, which can significantly reduce the weight of the target data source, which is consistent with the scenario that unstable data source should reduce resources; when p is at a medium level, a increases with p, but the growth rate is slower than p, so I1 = a / p increases with p, but the growth rate is gentle, which avoids that the target data source obtains too high weight at a medium successful rate; when p → 1 (the successful rate is extremely high), a ≈ 1, so I1 ≈ 1, which can assign a larger weight to the target data source, and does not additionally amplify the weight. Based on the formula, the non-linear adjustment is realized through the exponential function, the punishment for the low successful rate data source is more significant (a is much smaller than p), the reward for the high successful rate data source is more restrained (a is close to p), which avoids that the resources are tilted to extremely unstable or excessively compatible data sources.

[0115] In S642, a ratio of the latest successful crawling number of the target data source to the second dynamic coefficient is determined as a second collection power sub-index of the target data source; the second collection power sub-index is greater than or equal to 0 and less than or equal to 1.

[0116] In the embodiment, the second dynamic coefficient is a dynamic adjustment parameter related to the maximum value of the historical successful number of the target data source. As a specific embodiment, for the target data source, the second dynamic coefficient is determined according to the maximum value of the successful crawling number of the target data source in the target historical time period.

[0117] In the embodiment, the second collection power sub-index reflects the contribution of the successful number to the collection power index, and the second collection power sub-index is I2, I2 = B / β, B is the latest successful crawling number of the target data source, and the value range of the second collection power sub-index is [0, 1].

[0118] As a specific embodiment, β = w1 × B 1,max + (1-w1) × B 2,max . B 1,max is the maximum value of the successful crawling number of the target data source in a specified unit time period, and B 2,maxB1,max is the maximum value of the number of successful crawls of the target data source in the target historical time period excluding the specified unit time period from the target historical time period, the specified unit time period being the unit time period closest to the current time; w1 is a preset weight, and optionally, 0.5 < w1 < 1. Based on the formula, β integrates the best performance of the recent (B1,max) and the history (B2,max), and can avoid misjudgment caused by single time period data (such as short-term fluctuations or obsolete data), and w1 > 0.5 ensures that the recent performance weight is higher; β is updated with the historical performance of the data source, so that I2 can reflect the position of the current value of the data source relative to the historical best level, rather than the absolute value under a fixed benchmark; the essence of I2 = B / β is the ratio of the current performance of the data source to the historical best performance, when B is close to the recent or historical maximum value, I2→1 (indicating that the current performance is excellent); when B is much smaller than the historical maximum value, I2→0 (indicating that the current performance is poor), the embodiment measures the value of the current success number B through β, which not only focuses on the recent performance of the data source, but also takes into account its long-term stability, and finally makes I2 reflect the advantages and disadvantages of the current success number relative to the historical best level.

[0119] As a specific embodiment, the target historical time period includes a plurality of unit times, and the target data source is detected a plurality of times in any unit time, optionally, the target historical time period is 7 days, the unit time period is 1 day, and the target data source is detected once every 30 minutes (i.e., the detection period is 30 minutes), and each detection can obtain a number of successful crawls.

[0120] S643, acquiring the collection power index of the target data source according to the first collection power sub-index, the second collection power sub-index and the priority; the collection power index of the target data source is positively correlated with the first collection power sub-index, and the collection power index of the target data source is positively correlated with the second collection power sub-index.

[0121] In the embodiment, the priority reflects the importance of the data source, and the higher the importance, the greater the priority; the value range of the priority is [0, 1]. Optionally, the priority of the target data source is manually set.

[0122] As a specific embodiment, the product of the first collection power sub-index, the second collection power sub-index and the priority of the target data source is determined as the collection power index of the target data source.

[0123] The embodiment cooperatively evaluates the first collection power sub-index, the second collection power sub-index and the priority in three dimensions, and finally outputs an index (i.e., the collection power index) that comprehensively reflects the degree of value of the data source to be collected, avoiding the single dimension dominating the resource allocation, improving the comprehensiveness of the evaluation, and making the allocation of data collection resources more accurate, more dynamic and more in line with actual needs.

[0124] S650, determining the collection power of the target data source according to the collection power index of the target data source.

[0125] In this embodiment, the collection power index of the target data source is positively correlated with the collection power of the target data source. As a specific implementation, a mapping relationship between the collection power index and the collection power is established in advance, and the collection power of the target data source can be obtained by the above-mentioned pre-established mapping relationship on the basis of knowing the collection power index of the target data source.

[0126] In this embodiment, the collection power of the target data source reflects the collection intensity of the target data source, and the collection power of the target data source can be the collection frequency.

[0127] This embodiment periodically detects the target data source through a preset detection period, and calculates the latest crawling success number and success rate based on the latest preset detection result, which can capture the state change of the data source (such as anti-crawling strategy upgrade, server load capacity fluctuation, temporary failure, etc.) in real time. Compared with the lag response of the traditional fixed crawler strategy to the state change, this embodiment can keep the collection strategy synchronized with the current state of the target data source, and improve the adaptability of data collection to dynamic environment.

[0128] This embodiment calculates the collection power index based on the latest crawling success number (reflecting the current upper limit of the data source) and the latest crawling success rate (reflecting the current anti-crawling intensity), which can accurately match the real-time accessibility of the data source: for the data source in good state (high success rate, strong load capacity), higher collection power can be allocated to fully utilize its resources; for the data source in poor state (low success rate, frequent interception), the collection power is automatically reduced to reduce invalid requests (such as intercepted crawler resources, wasted bandwidth). This mechanism solves the problems of resource waste in poor state and insufficient collection in good state in the traditional fixed strategy, and improves the overall resource utilization efficiency.

[0129] This embodiment includes the target data source priority and the latest state index (i.e. the latest crawling success number and the latest crawling success rate) into the calculation of the collection power index, to ensure that the high-priority data source obtains reasonable resource tilt within its current state: when the high-priority data source is in good state, resources are preferentially allocated to ensure efficient collection of core data; when the high-priority data source is in poor state, avoid invalid consumption caused by blind investment of resources, and dynamically adjust based on its actual accessibility, balancing the priority weight and real-time feasibility, and improving the stability and reliability of core data collection.

[0130] Therefore, the embodiment reduces the dependence on manual intervention through the full-process automatic mechanism of periodic detection, automatic calculation of the latest state index, and dynamic generation of the collection power, is especially suitable for a multi-data-source and high-dynamic collection scenario, and significantly reduces the labor cost.

[0131] Embodiment Three

[0132] Compared with Embodiment Three, the embodiment further provides a process of determining the collection power of the target data source according to the collection power index of the target data source on the basis of Embodiment Two, and the process includes:

[0133] S651, obtaining a set of experimental crawler programs whose crawling results on the target data source are successful from an experimental collection group; the experimental collection group includes a plurality of experimental crawler programs.

[0134] In the embodiment, the experimental collection group is a set composed of a plurality of experimental crawler programs, and the experimental collection group includes different experimental crawler programs. The experimental collection group is used for preliminary crawling test on the candidate data sources in the candidate data source set, and the crawling results are obtained to evaluate the data source performance. The candidate data source set is a set of data sources (such as websites) to be evaluated, and is the target of data collection. The experimental crawler program is a program used to simulate the data collection process, which can initiate a crawling request to the data source and return a success or failure result. The candidate data source is a specific data source to be evaluated, and the target data source is any candidate data source in the candidate data source set, which is the object of evaluation in the embodiment. As a specific implementation, the target data source and the candidate data source are websites.

[0135] In the embodiment, the experimental crawler programs in the experimental collection group and the candidate data sources satisfy a preset condition, and the preset condition includes that the experimental crawler programs are compatible with the candidate data sources. The preset condition that the experimental crawler programs are compatible with the candidate data sources is a basic prerequisite for effective implementation of the experiment. If the crawler and the data source are not compatible, the crawler cannot normally crawl, a large amount of invalid failure data is generated, the set of experimental crawler programs whose crawling results are successful cannot be screened out, and the entire experimental process is stalled. Based on the preset condition that the experimental crawler programs are compatible with the candidate data sources, it can be ensured that all experimental crawlers can effectively access and crawl the candidate data sources, and invalid samples are avoided.

[0136] In this embodiment, the preset data source attributes of the candidate data sources in the candidate data source set are the same, and the preset data source attributes include at least one of the following attributes: data source type, anti-crawling strategy, page structure type, access protocol, and request frequency limit value. The data source type includes static website, dynamic website, etc., the anti-crawling strategy includes verification code verification, IP restriction, User-Agent detection, etc., the page structure type includes list type structure, detail type structure, nested type structure, irregular structure, etc., the access protocol includes HTTP, FTP, HTTPS, etc., and the request frequency limit value refers to the maximum number of requests allowed by the data source within a unit time. Therefore, the success rate of the experimental crawler program can be prevented from being distorted due to the attribute difference of the data source, the success rate of the experimental crawler program can truly reflect the characteristics of the data source itself, and the data sample for subsequently determining the crawling stability of the target data source according to the success rate of the experimental crawler program is improved without bias.

[0137] In this embodiment, the preset crawler program attributes of different experimental crawler programs in the experimental collection group are the same, and the preset crawler program attributes include at least one of the following attributes: crawling target, running parameter, and validity. The crawling target refers to specific data content that needs to be obtained from the data source, for example, title, publication time, and text of a news website, etc.; the running parameter refers to the configuration parameter of the crawler program, including the number of concurrent connections, timeout time, number of retries, etc.; and the validity refers to the functional integrity and correctness of the crawler program itself. The preset attributes of the experimental crawler programs are the same, which can exclude the influence of the differences of the crawlers on the experimental results, make the crawling ability and execution logic of all experimental crawler programs consistent, and make the difference in the success rate determined only by the real characteristics of the candidate data source, so that the crawling characteristic judgment index calculated based on the success rate is more objective, and truly reflects the stability of the target data source.

[0138] In this embodiment, each experimental crawler program in the experimental collection group is used to perform crawling test on each candidate data source in the candidate data source set, so that the crawling result of each experimental crawler program in the experimental collection group on each candidate data source in the candidate data source set can be obtained, the crawling result includes success and failure, and a set of experimental crawler programs whose crawling result on the target data source is success is determined as the experimental crawler program set.

[0139] S652, obtaining the crawling success rate of each experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set; the candidate data source set includes a plurality of candidate data sources.

[0140] In this embodiment, the crawling result of each experimental crawler program in the experimental crawler program set on each candidate data source in the candidate data source set is known, and on this basis, the crawling success rate of each experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set can be obtained. The crawling success rate of any experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set is the ratio of the number of successful crawls of the candidate data source in the candidate data source set by the experimental crawler program to the number of candidate data sources included in the candidate data source set.

[0141] S653, determining the crawling feature judgment index of the target data source according to the crawling success rate.

[0142] As a specific implementation, S653 includes:

[0143] S6531, obtaining the average value of the crawling success rate of the experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set.

[0144] S6532, obtaining the variance of the crawling success rate of the experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set.

[0145] S6533, determining the crawling feature judgment index of the target data source according to the average value and the variance of the crawling success rate; the crawling feature judgment index of the target data source is negatively correlated with the average value of the crawling success rate; and the crawling feature judgment index of the target data source is negatively correlated with the variance of the crawling success rate.

[0146] As a specific implementation, the average value and the variance of the crawling success rate are normalized respectively, then the difference between 1 and the normalized result of the average value is determined as a first index, and the difference between 1 and the normalized result of the variance is determined as a second index; the product of the first index and the second index is determined as the crawling feature judgment index of the target data source. Those skilled in the art know that normalization is a prior art, which will not be described here.

[0147] In this embodiment, the larger the crawling feature judgment index of the target data source is, the easier the target data source is to be crawled and the more stable the crawling difficulty is; otherwise, the target data source is more difficult to be crawled or the crawling difficulty is less stable.

[0148] S654, if the crawling feature judgment index of the target data source is greater than or equal to the preset judgment index threshold value, determining the collection power of the target data source according to a comparison result of the collection power index of the target data source and a preset collection power index threshold value; otherwise, determining the collection power of the target data source according to a comparison result of the collection power index of the target data source and an updated collection power index threshold value; the updated collection power index threshold value is obtained according to the preset collection power index threshold value, and the updated collection power index threshold value is not equal to the preset collection power index threshold value.

[0149] As a specific embodiment, the collection power index of the target data source is a comprehensive index quantitatively describing the intensity of resource investment and the degree of strategy aggressiveness when collecting data of the target data source, and the range of the collection power index of the target data source is [0, 1], which can be manually set.

[0150] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold value includes: if the collection power index of the target data source is greater than or equal to a preset first collection power index threshold value, determining the collection power of the target data source as a preset maximum collection power; if the collection power index of the target data source is less than or equal to a preset second collection power index threshold value, determining the collection power of the target data source as a preset minimum collection power; the preset first collection power index threshold value is greater than the preset second collection power index threshold value.

[0151] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold value further includes: if the collection power index of the target data source is less than the preset first collection power index threshold value and greater than the preset second collection power index threshold value, performing normalization processing on the collection power index of the target data source according to the preset first collection power index threshold value and the preset second collection power index threshold value, and determining the product of the result of the normalization processing on the collection power index of the target data source and the preset maximum collection power as the collection power of the target data source. It should be understood that the normalization processing here refers to maximum-minimum normalization processing.

[0152] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value includes: if the collection power index of the target data source is greater than or equal to the updated first collection power index threshold value, determining the collection power of the target data source as the preset maximum collection power; if the collection power index of the target data source is less than or equal to the updated second collection power index threshold value, determining the collection power of the target data source as the preset minimum collection power; the updated first collection power index threshold value is greater than the updated second collection power index threshold value; the updated first collection power index threshold value is less than the preset first collection power index threshold value, and the updated second collection power index threshold value is less than the preset second collection power index threshold value.

[0153] As a specific embodiment, determining the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value further includes: if the collection power index of the target data source is less than the updated first collection power index threshold value and greater than the updated second collection power index threshold value, performing normalization processing on the collection power index of the target data source according to the updated first collection power index threshold value and the updated second collection power index threshold value, and determining the larger value of the product of the result of the normalization processing on the collection power index of the target data source and the preset maximum collection power and the original collection power as the collection power of the target data source; wherein the original collection power is: the product of the result of the normalization processing on the collection power index of the target data source based on the preset first collection power index threshold value and the preset second collection power index threshold value and the preset maximum collection power. It should be understood that the normalization processing here refers to maximum-minimum normalization processing.

[0154] As a specific embodiment, T'1 is an updated first collection power index threshold, 0 < T'1 < 1, T'1 = (1 - k1 x (k - k0) / k) x T1, where T1 is a preset first collection power index threshold, 0 < T1 < 1, k1 is a preset first adjustment coefficient, 0 < k1 < 1, k is a preset determination index threshold, k0 is a crawling characteristic determination index of the target data source, and k0 < k. Optionally, T1, k1, and k are empirical values, for example, T1 = 0.8, k1 = 0.5, and k = 0.6. Based on the formula, when the actual crawling characteristic determination index of the target data source is less than the preset determination index threshold, the updated first collection power index threshold is less than the preset first collection power index threshold; and the more the actual crawling characteristic determination index of the target data source is less than the preset determination index threshold, the more the updated first collection power index threshold is less than the preset first collection power index threshold, which helps to enable the strong mode earlier to cope with possible crawling difficulties when the data source is unstable or difficult to be crawled.

[0155] As a specific embodiment, T'2 is an updated second collection power index threshold, 0 < T'2 < 1, T'2 = (1 - k2 x (k - k0) / k) x T2, where T2 is a preset second collection power index threshold, 0 < T2 < 1, k2 is a preset second adjustment coefficient, 0 < k2 < 1, k is a preset determination index threshold, k0 is a crawling characteristic determination index of the target data source, k0 < k, and (1 - k1) x T1 > (1 - k2) x T2. Optionally, T2 and k2 are empirical values, for example, T2 = 0.3 and k2 = 0.3. Based on the formula, the collection power index is reduced to a lower level before triggering the minimum power collection instruction, which forces the system to adopt a strategy of increasing the sampling power when the data source is unstable or difficult to be crawled, and is less likely to enter a low-power or sleep state, thereby giving priority to ensuring crawling success.

[0156] The embodiment obtains the crawling success rate of each experimental crawler program in the experimental crawler program set on the candidate data source in the candidate data source set, determines the crawling characteristic judgment index of the target data source according to the crawling success rate, and if the crawling characteristic judgment index of the target data source is greater than or equal to the preset judgment index threshold value, determines the collection power of the target data source according to the comparison result of the collection power index of the target data source and the preset collection power index threshold value; otherwise, determines the collection power of the target data source according to the comparison result of the collection power index of the target data source and the updated collection power index threshold value. Thus, the embodiment can intelligently distinguish the stability characteristics of the data source by introducing the crawling characteristic judgment index and constructing a dynamic threshold adjustment mechanism, and for the data source with stable characteristics, a conventional threshold is used for decision-making; for the data source with unstable characteristics, an updated threshold is used for decision-making; such differentiated processing method is conducive to improving the accuracy of power decision-making, thereby improving the success rate and efficiency of data collection as a whole.

[0157] Embodiment four

[0158] The embodiment provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0159] According to the number of times that the data of the target data source is accessed in the target time period, the data access popularity of the target data source is obtained; the target time period is a time period with the current time as the end point and a preset length.

[0160] According to the update frequency of the target data source in the target time period, the data update activity of the target data source is obtained.

[0161] According to the amount of data generated by collecting the target data source in the target time period, the output data amount index of the target data source is obtained.

[0162] According to the resources consumed by collecting the data of the target data source in the target time period, the data collection resource consumption index of the target data source is obtained.

[0163] According to the data access popularity, the data update activity, the output data amount index and the data collection resource consumption index of the target data source, the priority of the target data source is obtained.

[0164] According to the priority of the target data source, the data of the target data source is collected.

[0165] Embodiment five

[0166] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0167] The data access heat of the target data source is obtained according to the number of times that data of the target data source is accessed within a target time period; the target time period is a time period with a preset length ending at a current time.

[0168] The data update activity of the target data source is obtained according to the update frequency of the target data source within a target time period.

[0169] The output data amount index of the target data source is obtained according to the amount of data generated by the target data source within a target time period.

[0170] The data collection resource consumption index of the target data source is obtained according to the resources consumed by the data collection of the target data source within a target time period.

[0171] The priority of the target data source is obtained according to the data access heat, the data update activity, the output data amount index, and the data collection resource consumption index of the target data source.

[0172] The data of the target data source is collected according to the priority of the target data source.

[0173] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database, or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0174] While certain specific embodiments of the application have been described in detail herein for the purposes of exemplification, numerous other variations and modifications will be apparent to persons skilled in the art. Alterations and modifications of detail can be made by those skilled in the art, having the benefit of the above description, without departing from the spirit and scope of the application. It is intended that all such alterations and modifications be included within the scope of the application whose limits are to be determined only by the appended claims.

Claims

1. A data source acquisition method based on data source priority, characterized in that, The method comprises the following steps: According to the number of times of accessing the data of the target data source in a target time period, a data access heat of the target data source is obtained; the target time period is a time period with a preset length ending at a current time point; According to the update frequency of the target data source in the target time period, a data update activity of the target data source is obtained; According to the amount of data generated by collecting the target data source in the target time period, an output data amount index of the target data source is obtained; According to the resources consumed by collecting the data of the target data source in the target time period, a data collection resource consumption index of the target data source is obtained; According to the data access heat, the data update activity, the output data amount index and the data collection resource consumption index of the target data source, a priority of the target data source is obtained; According to the priority of the target data source, the data of the target data source is collected.

2. The data source acquisition method based on data source priority according to claim 1, characterized in that, According to the update frequency of the target data source in the target time period, a data update activity of the target data source is obtained, which comprises: According to the update frequency of the target data source in the target time period, an update frequency curve of the target data source in the target time period is obtained; the update frequency of the target data source in the target time period comprises the update frequency of the target data source corresponding to each unit time in the target time period; the horizontal axis of the update frequency curve of the target data source in the target time period is time, and the vertical axis of the update frequency curve of the target data source in the target time period is update frequency; An average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period is obtained; According to the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period and the average value of the update frequency of the target data source in different unit times in the target time period, a data update activity of the target data source is obtained.

3. The data source acquisition method based on data source priority according to claim 2, characterized in that, According to the average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period and the average value of the update frequency of the target data source in different unit times in the target time period, a data update activity of the target data source is obtained, which comprises: The average value of the absolute value of the slope of the update frequency curve of the target data source in the target time period is normalized to obtain a first activity of the target data source; The average value of the update frequency of the target data source in different unit times in the target time period is normalized to obtain a second activity of the target data source; According to the weight corresponding to the slope of the preset update frequency curve, the first activity of the target data source, the weight corresponding to the average value of the preset update frequency and the second activity of the target data source, a data update activity of the target data source is obtained.

4. The data source acquisition method based on data source priority according to claim 1, characterized in that, The data access heat of the target data source is positively correlated with the number of times of accessing the data of the target data source in the target time period; the output data amount index of the target data source is positively correlated with the amount of data generated by collecting the target data source in the target time period; The data collection resource consumption index of the target data source is positively correlated with the resources consumed by collecting the data of the target data source in the target time period.

5. The data source acquisition method based on data source priority according to claim 4, characterized in that, According to the data access heat, the data update activity, the output data amount index and the data collection resource consumption index of the target data source, a priority of the target data source is obtained, which comprises: obtaining a weight corresponding to data access heat, a weight corresponding to data update activity, a weight corresponding to output data volume, and a weight corresponding to data collection resource consumption; the weight corresponding to data access heat, the weight corresponding to data update activity, the weight corresponding to output data volume, and the weight corresponding to data collection resource consumption are all greater than 0; determining a product of the weight corresponding to data access heat and data access heat of the target data source as a first sub-priority of the target data source; determining a product of the weight corresponding to data update activity and data update activity of the target data source as a second sub-priority of the target data source; determining a product of the weight corresponding to output data volume and output data volume index of the target data source as a third sub-priority of the target data source; determining a product of the weight corresponding to data collection resource consumption and data collection resource efficiency of the target data source as a fourth sub-priority of the target data source; the data collection resource efficiency of the target data source is a difference between 1 and data collection resource consumption index of the target data source; determining a sum of the first sub-priority of the target data source, the second sub-priority of the target data source, the third sub-priority of the target data source, and the fourth sub-priority of the target data source as a priority of the target data source.

6. The data source acquisition method based on data source priority according to claim 5, characterized in that, obtaining a weight corresponding to data access heat, a weight corresponding to data update activity, a weight corresponding to output data volume, and a weight corresponding to data collection resource consumption includes: obtaining a target variable; the target variable is a manually labeled priority; taking four indexes of the target data source as characteristic variables to construct a data set containing a corresponding relationship between the characteristic variables and the target variable; the four indexes include data access heat, data update activity, output data volume index, and data collection resource efficiency; training a linear regression model based on the data set to obtain coefficients corresponding to each characteristic variable in the linear regression model; performing normalization processing on the coefficients, so that a sum of the normalized coefficients is 1 and each coefficient is greater than 0, and the normalized coefficients are respectively determined as the weight corresponding to data access heat, the weight corresponding to data update activity, the weight corresponding to output data volume, and the weight corresponding to data collection resource consumption.

7. The data source acquisition method based on data source priority according to claim 6, characterized in that, The linear regression model is trained by a least square method.

8. The data source acquisition method based on data source priority according to claim 1, characterized in that, According to the priority of the target data source, obtaining a collection power of the target data source, and collecting data of the target data source based on the collection power of the target data source; the collection power of the target data source is positively correlated with the priority of the target data source.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the data source collection method based on the priority of the data source according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the data source collection method based on the priority of the data source according to any one of claims 1 to 8.

Images