An artificial intelligence-based network data crawling and anomaly detection method

Abstract

This invention belongs to the field of network data processing technology, and specifically relates to an artificial intelligence-based method for network data crawling and anomaly detection. Addressing the shortcomings of existing technologies in network data processing, this invention first filters an initial URL list to remove invalid and low-value URLs. Then, it uses a model that combines the advantages of CNN and RNN to identify webpage data regions, generates extraction paths using a dynamic XPath algorithm, calculates fingerprints using a multi-granularity hash algorithm for fingerprint comparison, and finally combines improved DBN and IF algorithms to detect data anomalies. This invention can adapt to complex webpage structure changes, accurately crawl data, precisely identify data updates or additions, effectively detect abnormal patterns, and improve data processing efficiency, accuracy, and security, providing a reliable solution for network data processing.

Description

Technical Field

[0001] This invention belongs to the field of network data processing technology, and in particular relates to a method for network data capture and anomaly detection based on artificial intelligence. Background Technology

[0002] In today's era of digital information explosion, the internet has become a hub for massive amounts of data, playing a crucial role in numerous fields such as business intelligence, market research, and financial risk analysis. However, the complexity, diversity, and dynamic nature of online data pose significant challenges to its effective acquisition and accurate analysis. Existing data scraping technologies often rely on manually written rules or simple template matching, which may be effective for web page data with relatively fixed structures and standardized formats, but falls short when faced with increasingly complex and frequently updated web page structures. Different types of web pages exhibit significant differences in layout, tag usage, and element nesting, making traditional methods severely inadequate in data scraping. Simultaneously, in the field of anomaly detection, traditional statistical analysis methods and rule-based detection techniques also face numerous difficulties. Statistical analysis methods make certain assumptions about data distribution, while actual online data often exhibits complex non-normal distributions, making statistical analysis methods prone to high false positive rates. Rule-based detection methods require manually defining a large number of complex rules to identify anomalies, which is not only time-consuming and labor-intensive but also struggles to cope with constantly emerging new anomaly patterns. With the increasing diversification of cyberattack methods and the more covert methods of data tampering, higher demands are being placed on the accuracy and timeliness of data anomaly detection. Summary of the Invention

[0003] To address the technical problems existing in the background art described above, this invention proposes a method for network data crawling and anomaly detection based on artificial intelligence.

[0004] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0005] S1. First, filter the initial URL list to remove invalid and low-value URLs;

[0006] S2. Next, for different types of web pages, a web page structure recognition model based on deep learning is used. Based on the identified data areas, a dynamic XPath generation algorithm is used to generate path expressions for extracting data.

[0007] S3. Then, fingerprint calculation is performed on each captured data using a multi-granularity hash algorithm, and the fingerprint differences are compared.

[0008] S4. Finally, based on the combination of the improved Deep Belief Network (DBN) and the Isolation Forest (IF) algorithm, anomalies in the data are detected. The fused feature vector of the data to be detected is input into the trained anomaly detection model, and the data is judged to be abnormal according to the preset threshold.

[0009] The specific implementation of generating the path expression for data extraction using the dynamic XPath generation algorithm in step S2 is as follows:

[0010] S21. First, for each element in the webpage, extract the HTML semantic tags, extract the keyword vectors of the text within the element, and convert the semantic tags into vector form;

[0011] S22. For any two elements and Calculate the semantic association strength: ,in It is the similarity of semantic label vectors. It is the similarity of keyword vectors. These are weighting coefficients;

[0012] S23, Using web page elements as nodes A directed graph is constructed using semantic association strength as edge weights;

[0013] S24. Then find the shortest path, with the root node r and the target node t. The shortest distance from root node r to node v is initially... For other nodes , For a set S of nodes where the shortest path has been determined, initially... For a node u that is not in S, update its shortest distance: ,in This represents the weight of the edge from node v to node u; then the process is repeated until the target node t is added to S, generating the initial XPath expression;

[0014] S25. Next, path optimization is performed, and the risk value of structural changes and the probability matrix of webpage structural changes are calculated. ,in Indicates from node To the node The probability of structural change for each node pair in path P. Calculate structural risk value : ,in It is a node The depth in the webpage structure is calculated from the root node to... The number of node layers traversed determines the process.

[0015] S26. Finally, perform path adjustments, for each node in path P... ,like If the value is greater than a preset threshold, then in the semantic relationship graph G, search for nodes... Nodes with similar semantics and low risk of structural change Replace the nodes in path P to obtain the optimized XPath expression.

[0016] Preferably, the webpage structure recognition model based on deep learning used in step S2 adopts a hybrid architecture that combines the advantages of CNN and RNN. The CNN layer is used to extract local features of the webpage, and performs preliminary extraction of features of various HTML tags, text blocks, and image region elements in the webpage. The convolution kernel slides on the HTML structure text representation of the webpage to capture feature patterns at different positions and scales. The RNN layer is used to learn the sequence relationship and contextual information between these local features.

[0017] Preferably, the implementation of the hybrid architecture that combines the advantages of CNN and RNN is as follows:

[0018] Step 1: First, the HTML structure of the webpage is represented as H, and the i-th convolutional kernel of the CNN layer is represented as... Then the local feature map F output by the CNN layer is: Where m is the number of convolution kernels, , This represents the feature vector in the local feature map F obtained after CNN processing, with values ​​ranging from 1 to m. For convolution operations, For bias terms;

[0019] Step 2: Then, the local feature map F output by the CNN layer is input into the RNN layer to update the current hidden state and learn the sequence relationship between the features of web page elements;

[0020] Step 3: Next, design the loss function L, which is obtained by summing the webpage structure recognition loss, element classification loss, and structural similarity judgment loss;

[0021] Step 4: Finally, adversarial training is introduced to construct a discriminator network D that outputs a probability value, representing the probability that the input feature representation is true. The objective function of adversarial training is: ,in The distribution representing the structural features of real web pages. Here, G is the noise distribution, x is the generator network, and z is the real sample.

[0022] Preferably, in step S3, for the different differences in fingerprints, it is determined whether it is data update or new data. For data update, only the changed data is updated. For new data, it is processed according to the established data storage and processing procedures.

[0023] Preferably, step S3 uses a multi-granularity hash algorithm for fingerprint calculation, and the specific implementation of comparing fingerprint differences is as follows:

[0024] S31. First, the captured complete data set is treated as a whole. SHA-256 is used to perform hash calculation on the entire data set to obtain the feature fingerprint at the overall data level. ;

[0025] S32. Based on the data structure, divide the data into multiple data blocks. For each data block, calculate its hash value using the same SHA-256 algorithm to obtain the feature fingerprint corresponding to each data block. ;

[0026] S33. Next, within the data block, for the title elements, image elements, and link elements in the webpage that require special attention, the same hashing process is performed again to generate element-level feature fingerprints. ;

[0027] S34. When the newly captured data is used to calculate the overall data-level feature fingerprint... Then, it is compared with the overall data-level feature fingerprints corresponding to historical data in the fingerprint database. To make a comparison, if Continue to confirm by comparing data block level and element level. A block-level comparison was performed, and the data was marked as showing significant changes.

[0028] Preferably, step S4 is based on a combination of an improved deep belief network (DBN) and an isolated forest (IF) algorithm to detect data anomalies. The specific implementation for detecting data anomalies is as follows:

[0029] S41. First, design a multi-layer deep belief network, use a large number of normal data samples to perform unsupervised pre-training of DBN layer by layer, and use the feature vector output from the highest hidden layer of the pre-trained DBN as input data to construct an isolated forest.

[0030] S42. Next, define a joint loss function that combines the reconstruction error of DBN with the outlier score of Isolation Forest: ,in Let T be the reconstruction error of the Deep Belief Network (DBN), and T be the number of isolated trees. For each tree in an isolated forest, assign anomaly scores to data points;

[0031] S43. Input normal data and a small amount of labeled abnormal data into the model, calculate the loss according to the joint loss function, and adjust the weights and biases of DBN and related parameters in the isolated forest, including the construction parameters of the isolated tree and the parameters for calculating the anomaly score, through the backpropagation algorithm.

[0032] S44. Then, anomaly detection is performed. The data to be detected is preprocessed by inputting the feature vector extracted by DBN into the isolated forest and calculating the anomaly score of each isolated tree for that data point. The final anomaly score is obtained by combining the scores of all trees.

[0033] S45. Finally, based on the pre-set abnormal score threshold, it is determined whether the data point is abnormal data.

[0034] Preferably, in step S26, the node is searched Nodes with similar semantics and low risk of structural change The specific implementation is to adjust the semantic association strength. and structural risk value Combined, the calculation formula is: .

[0035] Compared with existing technologies, the advantages and positive effects of this invention are as follows: a webpage structure recognition model that integrates the strengths of CNN and RNN accurately extracts features and understands structural relationships; a dynamic XPath generation algorithm adapts to changes in webpages; a multi-granularity hashing algorithm accurately determines data updates and additions; and an improved combination of DBN and IF algorithms effectively detects anomalies. These innovations improve the accuracy and adaptability of data crawling, reduce the false positive rate of anomaly detection, provide a more efficient, accurate, and reliable solution for network data processing, and strongly promote the development of related fields. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the structure of the algorithm. Figure 2 A flowchart of the dynamic XPath generation algorithm. Detailed Implementation

[0038] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0039] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0040] In this embodiment, the effective acquisition and accurate analysis of web data are crucial in many fields during today's digital information age. However, existing technologies have many shortcomings when facing complex and ever-changing webpage structures and diverse data anomalies. This embodiment aims to elaborate on an artificial intelligence-based web data crawling and anomaly detection method to solve the problems of adaptability, accuracy, and efficiency of traditional methods. The structure and flow of this application are as follows: Figure 1 As shown.

[0041] To remove invalid and low-value URLs and improve data crawling efficiency, the invention first filters the initial URL list. In real-world network environments, many URLs may point to invalid pages or contain information meaningless for analysis, wasting network resources and crawling time, and potentially introducing noisy data. A large amount of labeled URL data is collected, including the URLs themselves and annotations indicating whether the corresponding pages contain the target data. For each URL, its feature information is extracted, such as page title, keyword tags, keywords in the URL, and HTML structure features of the page. These features serve as input to a machine learning classification algorithm. A support vector machine (SVM) classification algorithm is selected and trained using the extracted URL features and labeled data. During training, the algorithm learns the relationship between URL features and whether a page contains the target data. The trained model is evaluated using methods such as cross-validation, calculating metrics such as accuracy, recall, and F1 score. Each URL in the generated initial URL list is input into the trained classification model, which predicts whether its corresponding page is likely to contain the target data based on the URL's features. URLs predicted not to contain the target data are removed from the initial URL list.

[0042] To address the issue of significant differences in webpage structure across various types and the insufficient adaptability of traditional methods, this invention employs a hybrid architecture webpage structure recognition model that integrates the advantages of CNN and RNN. The CNN layer extracts local features from the webpage, performing preliminary feature extraction on various HTML tags, text blocks, and image region elements. Convolutional kernels slide across the HTML structural text representation of the webpage, capturing feature patterns at different locations and scales. The RNN layer learns the sequence relationships and contextual information between these local features. First, the HTML structural text representation of the webpage is denoted as H, and the i-th convolutional kernel of the CNN layer is represented as... Then the local feature map F output by the CNN layer is: Where m is the number of convolution kernels, , This represents the feature vector in the local feature map F obtained after CNN processing, with values ​​ranging from 1 to m. For convolution operations, The bias term is used; then the local feature map F output from the CNN layer is input into the RNN layer to update the current hidden state and learn the sequence relationship between web page element features; next, a loss function L is designed, which is obtained by summing the web page structure recognition loss, element classification loss, and structure similarity judgment loss; finally, adversarial training is introduced, and a discriminator network D is constructed to output a probability value, representing the probability that the input feature representation is true. The objective function of adversarial training is: ,in The distribution representing the structural features of real web pages. Here, G is the noise distribution, x is the generator network, and z is the real sample.

[0043] Based on this, a dynamic XPath generation algorithm is used to generate path expressions for extracting data, such as... Figure 2 As shown. First, for each element in the webpage, extract the HTML semantic tags, extract the keyword vectors of the text within the element, and convert the semantic tags into vector form; for any two elements... and Calculate the semantic association strength: ,in It is the similarity of semantic label vectors. It is the similarity of keyword vectors. It is a weighting coefficient; based on webpage elements as nodes. A directed graph is constructed using semantic association strength as edge weights; then, the shortest path is found, with the root node being r and the target node being t. The shortest distance from root node r to node v is initially... For other nodes , For a set S of nodes where the shortest path has been determined, initially... For a node u that is not in S, update its shortest distance: ,in This represents the weight of the edge from node v to node u; then the process is repeated until the target node t is added to S, generating the initial XPath expression; next, path optimization is performed, calculating the structural change risk value and the webpage structural change probability matrix. ,in Indicates from node To the node The probability of structural change for each node pair in path P. Calculate structural risk value : ,in It is a node The depth in the webpage structure is calculated from the root node to... The path is determined by the number of node layers traversed; finally, path adjustment is performed, for each node pair in path P. ,like If the value is greater than a preset threshold, then in the semantic relationship graph G, search for nodes... Nodes with similar semantics and low risk of structural change Calculated as the semantic association strength and structural risk value Combination: The nodes in path P are replaced to obtain the optimized XPath expression. This method allows the data extraction path to adapt to dynamic changes in the webpage structure. Even if the webpage layout or elements change, the target data can still be accurately extracted, greatly improving the accuracy and stability of data crawling.

[0044] To accurately identify data updates and additions, this invention employs a multi-granularity hash algorithm for fingerprint calculation and comparison. For different differences in the fingerprint, it determines whether the data is updated or added. For updates, only the changed data is updated; for additions, they are processed according to a predetermined data storage and processing flow. The specific implementation of the multi-granularity hash algorithm for fingerprint calculation and comparison is as follows: First, the captured complete dataset is treated as a whole, and SHA-256 is used to perform hash calculation on the entire dataset to obtain the feature fingerprint at the overall data level. Based on the data structure, the data is divided into multiple data blocks. For each data block, the same SHA-256 hash is used to calculate its hash value, thus obtaining the feature fingerprint corresponding to each data block. Next, within the data block, for key elements such as title elements, image elements, and link elements on the webpage, the same hashing process is performed again to generate element-level feature fingerprints. When newly captured data is used to calculate overall data-level feature fingerprints Then, it is compared with the overall data-level feature fingerprints corresponding to historical data in the fingerprint database. To make a comparison, if Continue to confirm by comparing data block level and element level. A block-level comparison was performed, and the data was marked as showing significant changes.

[0045] Finally, an improved Deep Belief Network (DBN) and the Isolation Forest (IF) algorithm are combined to detect anomalies in the data. The fused feature vector of the data to be detected is input into the trained anomaly detection model, and the model determines whether the data is abnormal based on a preset threshold. First, a multi-layered Deep Belief Network is designed, and the DBN is pre-trained layer by layer in unsupervised manner using a large number of normal data samples. The feature vector output from the highest hidden layer of the pre-trained DBN is then used as input data to construct the Isolation Forest. Next, a joint loss function is defined to combine the reconstruction error of the DBN with the anomaly score of the Isolation Forest. ,in Let T be the reconstruction error of the Deep Belief Network (DBN), and T be the number of isolated trees. To assign anomaly scores to each tree in the isolated forest for a data point, normal data and a small number of labeled anomaly data are input into the model. The loss is calculated using the joint loss function, and the weights and biases of the DBN and relevant parameters in the isolated forest, including the construction parameters of the isolated trees and the anomaly score calculation parameters, are adjusted using the backpropagation algorithm. Then, anomaly detection is performed. The data to be detected is preprocessed, and the feature vectors extracted by the DBN are input into the isolated forest to calculate the anomaly score for each isolated tree for that data point. The scores of all trees are combined to obtain the final anomaly score; finally, based on a pre-set anomaly score threshold, it is determined whether the data point is an anomalous. This combined approach fully leverages the advantages of DBN in feature learning and the efficiency of Isolation Forest in anomaly detection, effectively reducing the false positive rate and promptly and accurately identifying various complex anomaly patterns, providing strong protection for the security and reliability of network data.

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for web data crawling and anomaly detection based on artificial intelligence, characterized in that, Includes the following steps: S1. First, filter the initial URL list to remove invalid and low-value URLs; S2. Next, for different types of web pages, a web page structure recognition model based on deep learning is used. Based on the identified data areas, a dynamic XPath generation algorithm is used to generate path expressions for extracting data. S3. Then, fingerprint calculation is performed on each captured data using a multi-granularity hash algorithm, and the fingerprint differences are compared. S4. Finally, based on the combination of the improved Deep Belief Network (DBN) and the Isolation Forest (IF) algorithm, anomalies in the data are detected. The fused feature vector of the data to be detected is input into the trained anomaly detection model, and the data is judged to be abnormal according to the preset threshold. The specific implementation of generating the path expression for data extraction using the dynamic XPath generation algorithm in step S2 is as follows: S21. First, for each element in the webpage, extract the HTML semantic tags, extract the keyword vectors of the text within the element, and convert the semantic tags into vector form; S22. For any two elements and Calculate the semantic association strength: ,in It is the similarity of semantic label vectors. It is the similarity of keyword vectors. These are weighting coefficients; S23, Using web page elements as nodes A directed graph is constructed using semantic association strength as edge weights; S24. Then find the shortest path, with the root node r and the target node t. The shortest distance from root node r to node v is initially... For other nodes , ; For a set S of nodes where the shortest path has been determined, initially... For a node u that is not in S, update its shortest distance: ,in This represents the weight of the edge from node v to node u; Then repeat the process until the target node t is added to S, generating the initial XPath expression; S25. Next, path optimization is performed, and the risk value of structural changes and the probability matrix of webpage structural changes are calculated. ,in Indicates from node To the node The probability of structural change for each node pair in path P. Calculate structural risk value : ,in It is a node The depth in the webpage structure is calculated from the root node to... The number of node layers traversed determines the process. S26. Finally, perform path adjustments, for each node in path P... ,like If the value is greater than a preset threshold, search for nodes in the semantic relationship graph G. Nodes with similar semantics and low risk of structural change Replace the nodes in path P to obtain the optimized XPath expression.

2. The method for network data crawling and anomaly detection based on artificial intelligence according to claim 1, characterized in that, The deep learning-based webpage structure recognition model used in step S2 employs a hybrid architecture that combines the advantages of CNN and RNN. The CNN layer is used to extract local features of the webpage, performing preliminary extraction of features from various HTML tags, text blocks, and image area elements in the webpage. The convolution kernel slides across the HTML structure text representation of the webpage to capture feature patterns at different positions and scales. The RNN layer is used to learn the sequence relationships and contextual information between these local features.

3. The method for network data crawling and anomaly detection based on artificial intelligence according to claim 2, characterized in that, The implementation of the hybrid architecture that combines the advantages of CNN and RNN is as follows: Step 1: First, the HTML structure of the webpage is represented as H, and the i-th convolutional kernel of the CNN layer is represented as... Then the local feature map F output by the CNN layer is: Where m is the number of convolution kernels, , This represents the feature vector in the local feature map F obtained after CNN processing, with values ​​ranging from 1 to m. For convolution operations, For bias terms; Step 2: Then, the local feature map F output by the CNN layer is input into the RNN layer to update the current hidden state and learn the sequence relationship between the features of web page elements; Step 3: Next, design the loss function L, which is obtained by summing the webpage structure recognition loss, element classification loss, and structural similarity judgment loss; Step 4: Finally, adversarial training is introduced to construct a discriminator network D that outputs a probability value, representing the probability that the input feature representation is true. The objective function of adversarial training is: ,in The distribution representing the structural features of real web pages. Here, G is the noise distribution, x is the generator network, and z is the real sample.

4. The method for network data crawling and anomaly detection based on artificial intelligence according to claim 1, characterized in that, In step S3, for the different differences in the fingerprint, it is determined whether it is data update or new data. For data update, only the changed data is updated. For new data, it is processed according to the established data storage and processing procedures.

5. The method for network data crawling and anomaly detection based on artificial intelligence according to claim 1, characterized in that, In step S3, a multi-granularity hash algorithm is used to calculate the fingerprint. The specific implementation method for comparing fingerprint differences is as follows: S31. First, the captured complete data set is treated as a whole. SHA-256 is used to perform hash calculation on the entire data set to obtain the feature fingerprint at the overall data level. ; S32. Based on the data structure, divide the data into multiple data blocks. For each data block, calculate its hash value using the same SHA-256 algorithm to obtain the feature fingerprint corresponding to each data block. ; S33. Next, within the data block, for the title elements, image elements, and link elements in the webpage that require special attention, the same hashing process is performed again to generate element-level feature fingerprints. ; S34. When the newly captured data is used to calculate the overall data-level feature fingerprint... Then, it is compared with the overall data-level feature fingerprints corresponding to historical data in the fingerprint database. To make a comparison, if Continue to confirm by comparing data block level and element level. A block-level comparison was performed, and the data was marked as showing significant changes.

6. The method for network data crawling and anomaly detection based on artificial intelligence according to claim 1, characterized in that, Step S4, based on a combination of an improved Deep Belief Network (DBN) and the Isolation Forest (IF) algorithm, is used to detect anomalies in the data. The specific implementation for detecting anomalies in the data is as follows: S41. First, design a multi-layer deep belief network, use a large number of normal data samples to perform unsupervised pre-training of DBN layer by layer, and use the feature vector output from the highest hidden layer of the pre-trained DBN as input data to construct an isolated forest. S42. Next, define a joint loss function that combines the reconstruction error of DBN with the outlier score of Isolation Forest: ,in Let T be the reconstruction error of the Deep Belief Network (DBN), and T be the number of isolated trees. For each tree in an isolated forest, assign anomaly scores to data points; S43. Input normal data and a small amount of labeled abnormal data into the model, calculate the loss according to the joint loss function, and adjust the weights and biases of DBN and related parameters in the isolated forest, including the construction parameters of the isolated tree and the parameters for calculating the anomaly score, through the backpropagation algorithm. S44. Then, anomaly detection is performed. The data to be detected is preprocessed by inputting the feature vector extracted by DBN into the isolated forest and calculating the anomaly score of each isolated tree for that data point. The final anomaly score is obtained by combining the scores of all trees. S45. Finally, based on the pre-set abnormal score threshold, it is determined whether the data point is abnormal data.

7. The method for network data crawling and anomaly detection based on artificial intelligence according to claim 1, characterized in that, In step S26, the node is searched. Nodes with similar semantics and low risk of structural change The specific implementation is to adjust the semantic association strength. and structural risk value Combined, the calculation formula is: .

Images