An adaptive data acquisition method and device

Abstract

This invention discloses an adaptive data acquisition method and apparatus. The method includes acquiring source code information of a webpage to be processed, a set of historical webpage source code, and a set of crawler information. The set of historical webpage source code includes several sets of historical webpage source code information; the set of crawler information includes several sets of crawler program information. The set of historical webpage source code and the set of crawler information are fused to obtain crawler strategy generation model information. The crawler strategy generation model information is used to process the source code information of the webpage to be processed to obtain webpage acquisition information. Therefore, this embodiment can adaptively adjust the acquisition strategy according to the actual situation, enabling efficient and accurate data acquisition, thereby reducing the daily maintenance costs for technicians and improving the efficiency and quality of data acquisition.

Description

Technical Field

[0001] This invention relates to the field of text data acquisition technology, and in particular to an adaptive data acquisition method and apparatus. Background Technology

[0002] With the rapid development of information technology, data acquisition plays a crucial role in numerous fields such as industrial monitoring, environmental monitoring, medical diagnosis, financial analysis, intelligent transportation, and smart agriculture. However, most existing data acquisition methods employ fixed acquisition frequencies and static acquisition strategies, lacking the ability to dynamically adapt to target data sources. This leads to problems such as resource waste, data redundancy, or the omission of key information. Especially in web page text data acquisition, due to differences in web page structure, content update frequency, and data value density, traditional methods often struggle to efficiently and accurately obtain the required information. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an adaptive data acquisition method and device that can adaptively adjust the acquisition strategy according to the actual situation, and can perform data acquisition efficiently and accurately, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0004] To address the aforementioned technical problems, a first aspect of the present invention discloses an adaptive data acquisition method, the method comprising:

[0005] S1, obtain the source code information of the webpage to be processed, the historical webpage source code set, and the crawler information set; the historical webpage source code set includes several historical webpage source code information; the crawler information set includes several crawler program information.

[0006] S2, the historical webpage source code set and the crawler information set are fused to obtain crawler strategy generation model information;

[0007] S3, using the crawler strategy to generate model information, process the source code information of the webpage to be processed to obtain webpage collection information.

[0008] As an optional implementation, in the first aspect of the present invention, the step of fusing the historical webpage source code set and the crawler information set to obtain crawler strategy generation model information includes:

[0009] S21, perform feature extraction processing on the historical webpage source code set and the crawler information set to obtain a webpage feature vector set and a crawler information feature vector set; the webpage feature vector set includes several webpage feature vectors; the crawler information feature vector set includes several crawler information feature vectors.

[0010] S22, Process the webpage feature vector set and the crawler information feature vector set to obtain the webpage training dataset;

[0011] S23, process the webpage training dataset to obtain crawler strategy generation model information.

[0012] As an optional implementation, in the first aspect of the present invention, the step of performing feature extraction processing on the historical webpage source code set and the crawler information set to obtain a webpage feature vector set and a crawler information feature vector set includes:

[0013] S211, Process the historical webpage source code set to obtain a webpage feature vector set;

[0014] S212, The crawler information set is processed to obtain a crawler information feature vector set.

[0015] As an optional implementation, in the first aspect of the present invention, processing the crawler information set to obtain a crawler information feature vector set includes:

[0016] S2121, perform an extraction operation on any crawler program information in the crawler information set to obtain the running frequency feature information and access depth feature information of the crawler program information;

[0017] S2122, Perform statistical processing on the crawler program information to obtain the data volume characteristic information and algorithm type characteristic information of the crawler program information;

[0018] S2123, quantize the running frequency feature information, the access depth feature information, the data volume feature information and the algorithm type feature information to obtain quantized feature information;

[0019] S2124, using a web crawler vectorization calculation model, the quantized feature information is transformed into a vector to obtain the web crawler information feature vector corresponding to the web crawler program information.

[0020] The web crawler vectorization calculation model is as follows:

[0021]

[0022] In the formula, v i Let QZ be the i-th component of the crawler information feature vector. i LH represents the weight value corresponding to the quantization information of the i-th feature in the quantization feature information. i LH j and LH kLet XG represent the quantization information of the i-th feature, the j-th feature, and the k-th feature, respectively, where α is the quantization feature information adjustment factor. i,j XG is the correlation coefficient between the i-th feature and the j-th feature in the quantized feature information. i,k Let N be the correlation coefficient between the i-th feature and the k-th feature in the quantized feature information, and N be the number of features in the quantized feature information.

[0023] As an optional implementation, in the first aspect of the present invention, processing the webpage training dataset to obtain crawler strategy generation model information includes:

[0024] S231, perform missing value processing on the web page training dataset to obtain the first training dataset;

[0025] S232, perform outlier processing on the first training dataset to obtain the second training dataset;

[0026] S233, using the second training dataset, train the initial model information of the crawling strategy to obtain the crawling strategy generation model information.

[0027] As an optional implementation, in a first aspect of the present invention, the step of performing outlier processing on the first training dataset to obtain a second training dataset includes:

[0028] S2321, Using a local sample density calculation model, the first training dataset is processed to obtain sample local density information; the sample local density information includes several sample local density values.

[0029] The local sample density calculation model is as follows:

[0030]

[0031] In the formula, ρ i3 YBB is the i-th local density value of the sample in the local density information of the sample. i3 and YBB j3 These are the i3rd and j3rd first training sample information in the first training dataset, respectively, where M3 is the number of first training sample information in the first training dataset, and ||·|| is the Euclidean norm.

[0032] S2322, averaging all the sample local density values ​​in the sample local density information to obtain the sample average density value;

[0033] S2323, The average density value of the sample and the preset density threshold coefficient are processed to obtain the sample density threshold;

[0034] S2324, default s=1;

[0035] S2325, determine whether the s-th local density value of the sample in the sample local density information is less than the sample density threshold, and obtain the first determination result;

[0036] When the first judgment result is negative, the information of the s-th first training sample in the first training dataset is added to the second training dataset;

[0037] When the first judgment result is yes, execute S2326;

[0038] S2326, determine whether s is greater than the number of first training sample information in the first training dataset, and obtain a second judgment result;

[0039] When the second judgment result is negative, increment s by 1 and execute S2324;

[0040] If the second judgment result is yes, execute S233.

[0041] As an optional implementation, in the first aspect of the present invention, the step of using the crawler strategy to generate model information and processing the source code information of the webpage to be processed to obtain webpage collection information includes:

[0042] S31, use the crawler strategy to generate model information, process the source code information of the webpage to be processed, and obtain the collection strategy encoding information;

[0043] S32, according to the acquisition strategy encoding information, the initial crawler program information is processed by parameter configuration to obtain the target crawler program information;

[0044] S33, using the target crawler program information, process the source code information of the webpage to be processed to obtain the collected data information;

[0045] S34, The collected data information is processed to obtain web page collection information.

[0046] A second aspect of this invention discloses an adaptive data acquisition device, the device comprising:

[0047] The acquisition module is used to acquire the source code information of the webpage to be processed, the historical webpage source code set, and the crawler information set; the historical webpage source code set includes several historical webpage source code information sets; the crawler information set includes several crawler program information sets.

[0048] The first calculation module is used to fuse the historical webpage source code set and the crawler information set to obtain crawler strategy generation model information.

[0049] The second calculation module is used to generate model information using the crawler strategy, process the source code information of the webpage to be processed, and obtain webpage collection information.

[0050] A third aspect of this invention discloses another adaptive data acquisition device, the device comprising:

[0051] processor;

[0052] A memory coupled to the processor stores executable program code;

[0053] The processor calls the executable program code stored in the memory to execute some or all of the steps of the adaptive data acquisition method disclosed in the first aspect of the present invention.

[0054] A fourth aspect of the present invention discloses a computer-readable storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps of the adaptive data acquisition method disclosed in the first aspect of the present invention.

[0055] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0056] In this embodiment of the invention, the source code information of the webpage to be processed, a set of historical webpage source code, and a set of crawler information are obtained. The set of historical webpage source code includes several sets of historical webpage source code information; the set of crawler information includes several sets of crawler program information. The set of historical webpage source code and the set of crawler information are fused to obtain crawler strategy generation model information. The crawler strategy generation model information is used to process the source code information of the webpage to be processed to obtain webpage collection information. Therefore, this embodiment can adaptively adjust the collection strategy according to the actual situation, enabling efficient and accurate data collection, thereby reducing the daily maintenance costs for technical personnel and improving the efficiency and quality of data collection. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1This is a flowchart illustrating an adaptive data acquisition method disclosed in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the structure of an adaptive data acquisition device disclosed in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of another adaptive data acquisition device disclosed in an embodiment of the present invention. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0063] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0064] This invention discloses an adaptive data acquisition method and apparatus, which can adaptively adjust the acquisition strategy according to the actual situation, enabling efficient and accurate data acquisition. This helps reduce the daily maintenance costs for technicians and improves the efficiency and quality of data acquisition. Detailed descriptions follow.

[0065] Example 1

[0066] Please see Figure 1 , Figure 1 This is a flowchart illustrating an adaptive data acquisition method disclosed in an embodiment of the present invention. Wherein, Figure 1The described adaptive data acquisition method is applied in adaptive data acquisition devices, such as local servers or cloud servers used for adaptive data acquisition optimization management, etc., and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the adaptive data acquisition method may include the following operations:

[0067] S1, obtain the source code information of the webpage to be processed, the historical webpage source code set, and the crawler information set; the historical webpage source code set includes several historical webpage source code information; the crawler information set includes several crawler program information.

[0068] It should be noted that the above-mentioned historical webpage source code collection is a webpage source code library: it collects no less than 100 representative webpage source code samples (historical webpage source code information), covering four major categories: e-commerce platforms (accounting for 40%), news portals (accounting for 30%), social networks (accounting for 20%), and government agencies (accounting for 10%); the crawler information collection is a crawler program library: each webpage source code corresponds to at least one verified and valid crawler program source code (crawler program information), forming more than 100 webpage-crawler matching pairs.

[0069] S2, the historical webpage source code set and the crawler information set are fused to obtain crawler strategy generation model information;

[0070] S3, using the crawler strategy to generate model information, process the source code information of the webpage to be processed to obtain webpage collection information.

[0071] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0072] In an optional embodiment, the step of fusing the historical webpage source code set and the crawler information set to obtain crawler strategy generation model information includes:

[0073] S21, perform feature extraction processing on the historical webpage source code set and the crawler information set to obtain a webpage feature vector set and a crawler information feature vector set; the webpage feature vector set includes several webpage feature vectors; the crawler information feature vector set includes several crawler information feature vectors.

[0074] S22, Process the webpage feature vector set and the crawler information feature vector set to obtain the webpage training dataset;

[0075] It should be noted that the above processing involves merging the webpage feature vector set and the crawler information feature vector set. For example, if the webpage feature vector set is [a,b,c] and the crawler information feature vector set is [d,e,f,g], then after the above merging process, the resulting webpage training dataset is [a,b,c,d,e,fg].

[0076] S23, process the webpage training dataset to obtain crawler strategy generation model information.

[0077] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0078] In another optional embodiment, the step of performing feature extraction processing on the historical webpage source code set and the crawler information set to obtain a webpage feature vector set and a crawler information feature vector set includes:

[0079] S211, Process the historical webpage source code set to obtain a webpage feature vector set;

[0080] S212, The crawler information set is processed to obtain a crawler information feature vector set.

[0081] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0082] In another optional embodiment, the processing of the historical webpage source code set to obtain a webpage feature vector set includes:

[0083] S2111, perform syntax parsing on any historical webpage source code information in the historical webpage source code set to obtain the DOM tree structure information corresponding to the historical webpage source code information;

[0084] It should be noted that the above syntax parsing and processing can be performed using Python's BeautifulSoup library, lxml library, or html5lib library. Specifically, this embodiment of the invention does not limit the specific implementation.

[0085] It should be noted that by parsing the DOM structure, the hierarchical structure of the webpage can be extracted. In subsequent solutions, the crawler program can dynamically adjust the XPath or CSS Selector according to changes in the webpage's DOM structure, thereby improving the success rate of crawling.

[0086] S2112, traverse the DOM tree structure information to obtain a hierarchical node list; the hierarchical node list includes several hierarchical node information; the hierarchical node information includes tag type information, node text information, attribute information, and hierarchical depth;

[0087] It should be noted that the tag type information, node text information, attribute information, and hierarchy depth mentioned above are as follows:

[0088] Tag type information: refers to the tag name of each HTML element in the webpage DOM tree, such as... 、 、 、 .

[0089] Node text information: refers to the plain text content contained in an HTML element, the actual readable text after removing HTML tags.

[0090] Attribute information refers to the attributes of HTML tags, such as id, class, href, src, etc.

[0091] Nesting depth refers to the nesting level of a DOM element in the HTML structure, that is, the distance from the root node to the element.

[0092] It should be noted that the above traversal can use either depth-first traversal or breadth-first traversal; specifically, the embodiments of the present invention do not limit the specific traversal.

[0093] It should be noted that by traversing, we can obtain tag type information, node text information, attribute information, and hierarchy depth. This information can be input into the initial model information of the subsequent crawling strategy, which can improve the accuracy of model training, thereby enabling the final trained model to effectively improve the accuracy of data collection.

[0094] S2113, perform path weight calculation on the hierarchical node list information to obtain XPath weight information; the XPath weight information includes several XPath weight values;

[0095] It should be noted that XPath weight calculation can optimize the selection of crawler paths and improve the success rate of subsequent data collection.

[0096] It should be noted that XPath weight information is a set of weight values ​​calculated based on the XPath path, used to measure the importance of different nodes in the webpage DOM structure.

[0097] S2114, Calculate and process the attribute information to obtain CSS strength information;

[0098] It should be noted that the processing of S2113 and S2114 above can be performed using Python's BeautifulSoup library, lxml library, or html5lib library. Specifically, this embodiment of the invention does not limit the specific processing.

[0099] CSS strength information mainly refers to the style hierarchy, weight, and stability of web page elements. The importance of web page elements can be measured by calculating attribute information.

[0100] It should be noted that relying solely on DOM structure and XPath for webpage feature extraction may lead to issues when the webpage structure undergoes slight changes (such as adding new features). The XPath fails, causing crawling to fail. CSS strength information can help measure the relative importance of web page elements, allowing the crawler to still identify the main content when the structure changes.

[0101] S2115, Using a text content feature calculation model, the node text information is calculated and processed to obtain content feature information;

[0102] The text content feature calculation model is as follows:

[0103]

[0104] In the formula, NR represents the content feature information, and wd j2 cp is the structural complexity value of the j2th sentence in the node text information. i2 cx is the frequency of the i2th word in the node text information. i2 β is the part-of-speech weight of the i2th word in the node text information, β is the sentence complexity coefficient, M2 is the number of sentences in the node text information, and N2 is the number of words in the node text information;

[0105] It should be noted that the sentence complexity coefficient can be set by the user or obtained from historical data. Specifically, this embodiment of the invention does not limit the specifics.

[0106] It should be noted that the text content feature calculation model is used to calculate the content feature information (NR) of the webpage node text, which is used to determine the complexity and importance of the text information of that node, thereby filtering out more valuable text content when extracting webpage features and optimizing crawler strategies.

[0107] It should be noted that the sentence complexity value reflects the syntactic complexity of a sentence, measuring whether the text contains many long sentences, clauses, modifying structures, etc. It can be calculated using Stanford NLP, spaCy, etc. Generally, its value ranges as follows: 0 to 1, general sentence; 0.1 to 0.3, simple sentence; 0.3 to 0.6, medium complexity; 0.6 to 1.0, high complexity.

[0108] It should be noted that the part-of-speech weights reflect the importance of different words in the text content. For example, the part-of-speech weights for various words are: nouns 1.0, verbs 0.9, adjectives 0.8, adverbs 0.6, prepositions 0.3, and pronouns 0.2. Specific details are not limited in the embodiments of this invention.

[0109] It should be noted that the sentence complexity coefficient ranges from [0.5, 2]. A value between [0.5, 1] ​​is suitable for ordinary texts such as news articles and blogs; a value between [1, 1.5] is suitable for complex texts such as academic papers and technical documents; and a value between [1.5, 2] is suitable for professional literature that requires in-depth analysis of long sentences. The sentence complexity coefficient determines the degree to which the complexity of sentence structure affects the content's feature information. If the sentence complexity coefficient is too high, the crawler may only focus on complex text and ignore simple but important information; if it is too low, the crawler may crawl meaningless short sentences.

[0110] S2116, normalize and concatenate the XPath weight information, the CSS strength information and the content feature information to obtain the webpage feature vector corresponding to the historical webpage source code information.

[0111] It should be noted that the above normalization process can be performed using Min-Max normalization. The above concatenation process combines different features together. For example, the normalized XPath weight information is represented as a1, the normalized CSS strength information is represented as a2, and the normalized content feature information is represented as a3. Then the concatenated webpage feature vector is [a1, a2, a3].

[0112] It should be noted that by normalization and concatenation, multiple different types of features can be integrated together, allowing the model to utilize structural information, style information, and text content information simultaneously, thereby improving the overall ability to grasp the features of a webpage.

[0113] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0114] In an optional embodiment, processing the crawler information set to obtain a crawler information feature vector set includes:

[0115] S2121, perform an extraction operation on any crawler program information in the crawler information set to obtain the running frequency feature information and access depth feature information of the crawler program information;

[0116] It should be noted that the above extraction operations can be performed using Python's Scrapy tool; however, the specific implementation of this invention is not limited thereto.

[0117] It should be noted that by incorporating frequency and access depth features into model training, the model can understand how different crawling strategies operate, thereby predicting better crawling strategies and improving the accuracy and efficiency of web page data collection.

[0118] Among them, the running frequency characteristic information reflects the number of access requests of the crawler per unit time, which is used to measure the crawling speed of the crawler, and the access depth characteristic information reflects the link jump levels that the crawler needs to go through from the homepage to the target page during the crawling process.

[0119] S2122, Perform statistical processing on the crawler program information to obtain the data volume characteristic information and algorithm type characteristic information of the crawler program information;

[0120] It should be noted that the above statistical processing can be performed using Python's Scrapy tool; however, the specific implementation of this invention is not limited thereto.

[0121] It should be noted that the data volume characteristics and algorithm type characteristics, combined with the previously mentioned running frequency characteristics and access depth characteristics, can be used to optimize crawling strategies and improve the success rate and adaptability of web page data collection.

[0122] Among them, the data volume characteristic information measures the amount of data acquired by the crawler within a certain time or in a certain crawling task, and the algorithm type characteristic information is used to identify the strategy used by the crawler (such as BFS / DFS, multithreading, asynchronous, etc.).

[0123] S2123, quantize the running frequency feature information, the access depth feature information, the data volume feature information and the algorithm type feature information to obtain quantized feature information;

[0124] It should be noted that quantization can be performed using Scikit-learn, Pandas, or NumPy; the specific implementation in this invention is not limited to any particular method. For categorical features, such as those from web crawling algorithms (BFS, DFS, Async, Multithread, etc.), they can be converted into numerical features using One-Hot encoding. For some non-numerical features, discretization may be necessary to convert continuous features into discrete features. For example, access depth can be discretized into different levels (e.g., shallow, medium, deep). Discretization helps handle discontinuous features and allows them to be combined with other features in the model. For numerical features, normalization can be used to scale the features to a specific range, typically [0,1] or [-1,1], so that different feature values ​​can be compared on the same scale.

[0125] Quantization is used to transform unstructured features at different scales into digital features that can be processed by machine learning models. These quantized features can then be effectively used as input to the model, ensuring that different features can participate in model training, optimization, and inference at the same scale.

[0126] S2124, using a web crawler vectorization calculation model, the quantized feature information is transformed into a vector to obtain the web crawler information feature vector corresponding to the web crawler program information.

[0127] The web crawler vectorization calculation model is as follows:

[0128]

[0129] In the formula, v i Let QZ be the i-th component of the crawler information feature vector. i LH represents the weight value corresponding to the quantization information of the i-th feature in the quantization feature information. i LH j and LH k Let XG represent the quantization information of the i-th feature, the j-th feature, and the k-th feature, respectively, where α is the quantization feature information adjustment factor. i,j XG is the correlation coefficient between the i-th feature and the j-th feature in the quantized feature information. i,k Let N be the correlation coefficient between the i-th feature and the k-th feature in the quantized feature information, and N be the number of features in the quantized feature information.

[0130] It should be noted that the value of the above-mentioned quantitative feature information adjustment factor can be set by the user or obtained from historical data. Specifically, the embodiments of the present invention do not limit it.

[0131] It should be noted that the quantization feature information adjustment factor mainly controls the relative importance of quantization feature information in the calculation. It can affect the model's output by adjusting the contribution of quantization features to the crawler information vector. By adjusting the quantization feature information adjustment factor, the influence of each feature in the final result can be balanced, thereby improving the model's prediction accuracy. Its value range is between (0,1).

[0132] It should be noted that the correlation coefficient is an indicator that measures the degree of correlation between two features, with a value ranging from -1 to 1. By using the correlation coefficient, we can identify which features have strong dependencies, and thus appropriately adjust their contributions in the feature vector calculation. Highly correlated features may introduce redundancy in the model, reducing computational efficiency; appropriate adjustments can help improve the model's accuracy and stability.

[0133] Weight value QZ i This represents the importance of the i-th feature in the entire feature set. Its main function is to assign different relative weights to different features, determining the contribution of each feature to the model. The larger the weight value, the stronger the influence of the feature. The weight value typically ranges from [0,1].

[0134] It should be noted that, through the above-mentioned web crawler vectorization calculation model, the original web crawler features are transformed into an efficient feature vector through quantization, weighting, and correlation adjustment, which can be used by the model for training or inference, thereby improving the accuracy of the model in data parsing, collection strategy generation, and dynamic adjustment.

[0135] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0136] In an optional embodiment, processing the webpage training dataset to obtain crawler strategy generation model information includes:

[0137] S231, perform missing value processing on the web page training dataset to obtain the first training dataset;

[0138] It should be noted that the above missing value handling can be performed using Python libraries such as pandas and missingno, or K-nearest neighbor filling. Specifically, this embodiment of the invention does not limit the specific handling.

[0139] S232, perform outlier processing on the first training dataset to obtain the second training dataset;

[0140] S233, using the second training dataset, train the initial model information of the crawling strategy to obtain the crawling strategy generation model information.

[0141] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0142] In an optional embodiment, the outlier processing of the first training dataset to obtain the second training dataset includes:

[0143] S2321, Using a local sample density calculation model, the first training dataset is processed to obtain sample local density information; the sample local density information includes several sample local density values.

[0144] The local sample density calculation model is as follows:

[0145]

[0146] In the formula, ρ i3 YBB is the i-th local density value of the sample in the local density information of the sample. i3 and YBB j3 These are the i3rd and j3rd first training sample information in the first training dataset, respectively, where M3 is the number of first training sample information in the first training dataset, and ||·|| is the Euclidean norm.

[0147] It should be noted that the local sample density calculation model is used to calculate the local density of each sample in the training dataset. Its purpose is to measure the distribution of data points around a sample, which helps to discover data clusters. By calculating the local density value of a sample, if the local density value is small, it indicates a sparse data region, or even an outlier (anomaly).

[0148] S2322, averaging all the sample local density values ​​in the sample local density information to obtain the sample average density value;

[0149] S2323, The average density value of the sample and the preset density threshold coefficient are processed to obtain the sample density threshold;

[0150] It should be noted that the preset density threshold coefficient is 0.2. The above processing multiplies the average sample density value by the preset density threshold coefficient to obtain the sample density threshold.

[0151] It should be noted that the sample density threshold is used to filter out abnormal samples, optimize training data, and improve the adaptability of the crawler strategy.

[0152] S2324, default s=1;

[0153] S2325, determine whether the s-th local density value of the sample in the sample local density information is less than the sample density threshold, and obtain the first determination result;

[0154] When the first judgment result is negative, the information of the s-th first training sample in the first training dataset is added to the second training dataset;

[0155] When the first judgment result is yes, execute S2326;

[0156] S2326, determine whether s is greater than the number of first training sample information in the first training dataset, and obtain a second judgment result;

[0157] When the second judgment result is negative, increment s by 1 and execute S2324;

[0158] If the second judgment result is yes, execute S233.

[0159] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0160] In an optional embodiment, the step of using the second training dataset to train the initial model information of the crawler strategy to obtain the crawler strategy generation model information includes:

[0161] S2331, Using the second training dataset, the initial model information of the crawler strategy is trained to obtain the crawler strategy training result information and the crawler strategy training model information;

[0162] It should be noted that the initial model information for the above crawling strategy is an LSTM model.

[0163] S2332, The training result information of the crawler strategy is calculated and processed to obtain the crawler strategy loss function value;

[0164] S2333, determine whether the value of the crawler strategy loss function is less than the preset crawler strategy loss function threshold, and obtain the third judgment result;

[0165] It should be noted that the preset threshold value of the crawler strategy loss function is between [0.01, 0.05]. Specifically, the embodiments of the present invention do not limit it.

[0166] When the third judgment result is negative, the crawler strategy training model information is determined to be the crawler strategy initial model information, and the model parameter information of the crawler strategy initial model information is updated, and S2331 is executed.

[0167] It should be noted that the above-mentioned update of the model parameter information of the initial model information of the crawling strategy is carried out by the backpropagation algorithm combined with the gradient descent method. The model parameter information includes forget gate weight, input gate weight, candidate cell state weight, output gate weight, learning rate and number of samples used for training, etc. The specific details are not limited in the embodiments of the present invention.

[0168] When the third judgment result is yes, the crawler strategy training model information is determined to be the crawler strategy generation model information.

[0169] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0170] In an optional embodiment, the step of calculating and processing the training result information of the crawler strategy to obtain the crawler strategy loss function value includes:

[0171] The crawler strategy loss function calculation model is used to calculate and process the training result information of the crawler strategy to obtain the crawler strategy loss function value;

[0172] The calculation model for the crawler strategy loss function is as follows:

[0173]

[0174] In the formula, SH is the loss function value of the crawler strategy, JG is the real label information corresponding to the training result information of the crawler strategy, ZZ is the training result information of the crawler strategy, M1 and N1 are the number of features in the training result information of the crawler strategy and the number of crawler strategy training result values ​​corresponding to each feature, respectively, and JG j1 i1 and ZZ j1 i1 δ1 represents the i1st real label value of the j1st feature in the real label information corresponding to the crawler strategy training result information and the i1st crawler strategy training result value of the j1st feature in the crawler strategy training result information, respectively, and δ1 is the first weight parameter.

[0175] It should be noted that the first weight parameter can be set by the user or obtained from historical data. Specifically, this embodiment of the invention does not limit the specific weight parameter.

[0176] It should be noted that the value of the first weight parameter is between [0,1], and it is used to balance prediction error and stability. When the first weight parameter is large, the model pays more attention to the relative error, which is suitable for cases where the data distribution is relatively unbalanced. When the first weight parameter is small, the model relies more on cross-entropy loss, which is suitable for cases where the data is relatively balanced.

[0177] It should be noted that calculating the model through the crawler strategy loss function can improve the model's generalization ability, enabling it to adapt to changes in web pages and optimize the model's accuracy in predicting crawler strategies.

[0178] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0179] In an optional embodiment, the step of using the crawler strategy to generate model information and processing the source code information of the webpage to be processed to obtain webpage collection information includes:

[0180] S31, use the crawler strategy to generate model information, process the source code information of the webpage to be processed, and obtain the collection strategy encoding information;

[0181] It should be noted that the data collection strategy encoding information refers to the structured crawling instructions generated by the crawler strategy generation model based on the source code information of the webpage to be processed. These instructions are used to guide the crawler program to efficiently access, parse, and extract webpage data.

[0182] The above process involves using the source code information of the webpage to be processed as input to the crawler strategy generation model information, performing calculations on the crawler strategy generation model information, and outputting the collection strategy encoding information.

[0183] S32, according to the acquisition strategy encoding information, the initial crawler program information is processed by parameter configuration to obtain the target crawler program information;

[0184] It should be noted that the initial crawler program information is obtained through acquisition. Initial crawler program information refers to a basic crawler program that has not been optimized or customized. It is usually a general crawler framework or template containing basic crawling logic, for example:

[0185] Basic request modules (such as requests or Scrapy for HTTP requests);

[0186] Data parsing modules (such as BeautifulSoup, lxml, or XPath parsing);

[0187] Data storage module (such as JSON, CSV, or database);

[0188] Exception handling mechanisms (such as timeout retries and IP rotation).

[0189] It should be noted that the above parameter configuration processing can be done using tools such as Jinja2 and ConfigParser. In particular, the embodiments of the present invention do not limit the specifics.

[0190] It should be noted that the above parameter configuration specifically addresses the initial crawler program information, which is a basic version of the crawler with general crawling capabilities but not optimized for specific web pages. The collection strategy encoding information provides crawling rules (such as access frequency, URL structure, parsing method, anti-crawling measures, etc.). Parameter configuration using the collection strategy encoding information adjusts the initial crawler program information to enable it to accurately collect web page data according to the strategy, ultimately yielding the target crawler program information.

[0191] S33, using the target crawler program information, process the source code information of the webpage to be processed to obtain the collected data information;

[0192] It should be noted that the above-mentioned collected data information is obtained by the target crawler program information from the source code information of the webpage to be processed.

[0193] S34, The collected data information is processed to obtain web page collection information.

[0194] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0195] In an optional embodiment, processing the collected data to obtain webpage collection information includes:

[0196] S341, The acquisition success rate is obtained by calculating and processing the acquired data using the acquisition success rate calculation model;

[0197] The model for calculating the success rate of data acquisition is as follows:

[0198]

[0199] In the formula, cg is the collection success rate, YX is the number of target fields successfully captured in the collected data, YXZ is the total number of target fields in the collected data, BL is the number of requests rejected by the target website in the collected data, QQZ is the total number of requests in the collected data, JZ is the number of dynamic web page elements successfully loaded in the collected data, DTZ is the minimum number of dynamic web page elements required for the page to display normally in the collected data; δ2, δ3 and δ4 are the second weight parameter, the third weight parameter and the fourth weight parameter, respectively.

[0200] It should be noted that the success rate calculation model is used to evaluate the current crawler's collection quality, determine the effectiveness of the crawler strategy, and make adaptive adjustments when necessary.

[0201] It should be noted that the value range of the second weight parameter is between [0.4, 0.6], the value range of the third weight parameter is between [0.2, 0.4], and the value range of the fourth weight parameter is between [0.2, 0.4]. Among them, when the value of the second weight parameter is large, more attention is paid to the completeness of data crawling (suitable for scenarios with high data accuracy requirements), when the value of the third weight parameter is large, more attention is paid to the anti-blocking ability of the crawler (suitable for websites with strong anti-crawling measures), and when the value of the fourth weight parameter is large, more attention is paid to page integrity (suitable for scenarios that need to crawl AJAX / JS rendered data).

[0202] It should be noted that the number of successfully crawled target fields refers to the number of data entries in the webpage data successfully collected by the crawler that match the definition of the target fields; the total number of target fields refers to the total number of target fields that should theoretically be collected, that is, the total number of data entries in the webpage that meet the rules; the number of requests rejected by the target website refers to the number of requests that the crawler rejected when accessing the webpage due to anti-crawling mechanisms (such as blocking IPs or blocking User-Agents); the total number of requests refers to the total number of all HTTP requests sent by the crawler, including successful and failed requests; the number of successfully loaded dynamic webpage elements refers to the number of JavaScript dynamic elements (such as data loaded by AJAX) that were successfully loaded during page rendering; and the minimum number of dynamic webpage elements required for normal page display refers to the minimum number of dynamic elements required for the page to fully display the data.

[0203] S342, determine whether the acquisition success rate is less than the preset acquisition success threshold, and obtain the fourth judgment result;

[0204] It should be noted that the preset successful acquisition threshold ranges from [0.9, 0.95], but the specific value is not limited in this embodiment of the invention.

[0205] If the fourth judgment result is yes, execute S31;

[0206] When the fourth judgment result is yes, it means that the current crawler strategy has failed to effectively collect web page data and the data integrity is not high enough. This indicates that the current web page has problems such as changes in web page structure, errors in crawling rules, or interference from anti-crawling mechanisms. It is necessary to execute S31 to optimize or correct the crawler strategy in order to achieve the purpose of adaptive data collection, ensure that the final collected data is complete enough, and reduce the cost of subsequent data cleaning and supplementary collection.

[0207] If the fourth judgment result is negative, the collected data information is determined to be web page collected information.

[0208] As can be seen, implementing the adaptive data acquisition method described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0209] Example 2

[0210] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an adaptive data acquisition device disclosed in an embodiment of the present invention. Figure 2 The described adaptive data acquisition device is applied in an adaptive data acquisition optimization system, such as a local server or cloud server for adaptive data acquisition, etc., and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the adaptive data acquisition device includes:

[0211] The acquisition module 201 is used to acquire the source code information of the webpage to be processed, the historical webpage source code set, and the crawler information set; the historical webpage source code set includes several historical webpage source code information sets; the crawler information set includes several crawler program information sets.

[0212] The first calculation module 202 is used to fuse the historical webpage source code set and the crawler information set to obtain crawler strategy generation model information.

[0213] The second calculation module 203 is used to generate model information using the crawler strategy, process the source code information of the webpage to be processed, and obtain webpage collection information.

[0214] As can be seen, the adaptive data acquisition device described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0215] Example 3

[0216] Please see Figure 3 , Figure 3 This is a schematic diagram of another adaptive data acquisition device disclosed in an embodiment of the present invention. Figure 3 The described adaptive data acquisition device is applied in an adaptive data acquisition optimization system, such as a local server or cloud server for adaptive data acquisition, etc., and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the adaptive data acquisition device includes:

[0217] Processor 301;

[0218] A memory 302 containing executable program code is coupled to the processor 301;

[0219] The processor 301 calls the executable program code stored in the memory 302 to execute some or all of the steps of the adaptive data acquisition method of Embodiment 1.

[0220] As can be seen, the adaptive data acquisition device described in the embodiments of the present invention can adaptively adjust the acquisition strategy according to the actual situation, and can efficiently and accurately acquire data, thereby helping to reduce the daily maintenance costs of technicians and improve the efficiency and quality of data acquisition.

[0221] Example 4

[0222] This invention discloses a computer-readable storage medium storing computer instructions. When the computer instructions are invoked, they are used to execute some or all of the steps of the adaptive data acquisition method of Embodiment 1.

[0223] Example 5

[0224] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform some or all of the steps in the adaptive data acquisition method described in Embodiment 1.

[0225] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0226] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0227] Finally, it should be noted that the adaptive data acquisition method and apparatus disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive data acquisition method, characterized in that, The method includes: S1, obtain the source code information of the webpage to be processed, the historical webpage source code set, and the crawler information set; the historical webpage source code set includes several historical webpage source code information; the crawler information set includes several crawler program information. S2, the historical webpage source code set and the crawler information set are fused to obtain crawler strategy generation model information; S3, using the crawler strategy to generate model information, process the source code information of the webpage to be processed to obtain webpage collection information; S2 includes: S21, perform feature extraction processing on the historical webpage source code set and the crawler information set to obtain a webpage feature vector set and a crawler information feature vector set; the webpage feature vector set includes several webpage feature vectors; the crawler information feature vector set includes several crawler information feature vectors. S22, Process the webpage feature vector set and the crawler information feature vector set to obtain the webpage training dataset; S23, Process the webpage training dataset to obtain crawler strategy generation model information, including: S231, perform missing value processing on the web page training dataset to obtain the first training dataset; S232, perform outlier processing on the first training dataset to obtain the second training dataset; S233, using the second training dataset, the initial model information of the crawler strategy is trained to obtain the crawler strategy generation model information; S232 includes: S2321, Using a local sample density calculation model, the first training dataset is processed to obtain sample local density information; the sample local density information includes several sample local density values. The local sample density calculation model is as follows: In the formula, ρ i3 YBB is the i-th local density value of the sample in the local density information of the sample. i3 and YBB j3 These are the i3rd and j3rd first training sample information in the first training dataset, respectively, where M3 is the number of first training sample information in the first training dataset, and ||·|| is the Euclidean norm. S2322, averaging all the sample local density values ​​in the sample local density information to obtain the sample average density value; S2323, The average density value of the sample and the preset density threshold coefficient are processed to obtain the sample density threshold; S2324, default s=1; S2325, determine whether the s-th local density value of the sample in the sample local density information is less than the sample density threshold, and obtain the first determination result; When the first judgment result is negative, the information of the s-th first training sample in the first training dataset is added to the second training dataset; When the first judgment result is yes, execute S2326; S2326, determine whether s is greater than the number of first training sample information in the first training dataset, and obtain a second judgment result; When the second judgment result is negative, increment s by 1 and execute S2324; If the second judgment result is yes, execute S233.

2. The adaptive data acquisition method according to claim 1, characterized in that, The step of performing feature extraction processing on the historical webpage source code set and the crawler information set to obtain a webpage feature vector set and a crawler information feature vector set includes: S211, Process the historical webpage source code set to obtain a webpage feature vector set; S212, The crawler information set is processed to obtain a crawler information feature vector set.

3. The adaptive data acquisition method according to claim 2, characterized in that, The process of processing the crawler information set to obtain a crawler information feature vector set includes: S2121, extract any crawler program information in the crawler information set to obtain the running frequency feature information and access depth feature information of the crawler program information; S2122, Perform statistical processing on the crawler program information to obtain the data volume characteristic information and algorithm type characteristic information of the crawler program information; S2123, quantize the running frequency feature information, the access depth feature information, the data volume feature information and the algorithm type feature information to obtain quantized feature information; S2124, using a web crawler vectorization calculation model, the quantized feature information is transformed into a vector to obtain the web crawler information feature vector corresponding to the web crawler program information. The web crawler vectorization calculation model is as follows: In the formula, v i Let QZ be the i-th component of the crawler information feature vector. i and QZ j These are the weight values ​​corresponding to the quantization information of the i-th and j-th features in the quantization feature information, respectively, LH i LH j and LH k Let XG represent the quantization information of the i-th feature, the j-th feature, and the k-th feature, respectively, where α is the quantization feature information adjustment factor. i,j XG is the correlation coefficient between the i-th feature and the j-th feature in the quantized feature information. i,k Let N be the correlation coefficient between the i-th feature and the k-th feature in the quantized feature information, and N be the number of features in the quantized feature information.

4. The adaptive data acquisition method according to claim 1, characterized in that, The process of generating model information using the crawler strategy, processing the source code information of the webpage to be processed, and obtaining webpage collection information includes: S31, use the crawler strategy to generate model information, process the source code information of the webpage to be processed, and obtain the collection strategy encoding information; S32, according to the acquisition strategy encoding information, the initial crawler program information is processed by parameter configuration to obtain the target crawler program information; S33, using the target crawler program information, process the source code information of the webpage to be processed to obtain the collected data information; S34, process the collected data information to obtain web page collection information.

5. An adaptive data acquisition device, characterized in that, The device includes: processor; A memory coupled to the processor stores executable program code; The processor calls the executable program code stored in the memory to execute the adaptive data acquisition method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when invoked, are used to execute the adaptive data acquisition method as described in any one of claims 1-4.

Images