Internet data collection method and system based on data analysis

Abstract

The application discloses an internet data collection method and system based on data analysis, and belongs to the technical field of big data analysis. The method comprises the following steps: constructing semantic constraint representation of a target data field; performing real-time feature comparison and outputting a feature comparison result; performing hierarchical isolation processing on the current data field based on the feature comparison result, monitoring the data increase rate in the abnormal isolation area, and updating the initial collection rule; identifying polluted data records, generating counterfactual compensation data, and performing incremental parameter compensation processing on a downstream processing node receiving current data records using the counterfactual compensation data. The application solves the problem that semantic misplacement data caused by page structure evolution is difficult to identify and pollutes downstream models by constructing semantic constraint representation of a target data field, performing real-time feature comparison, and using counterfactual compensation data to implement incremental parameter compensation on downstream processing nodes.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, and in particular to a method and system for collecting internet data based on data analytics. Background Technology

[0002] With the explosive growth of internet information, data collection technology based on data analysis has become a fundamental support for fields such as public opinion monitoring, knowledge graph construction, and e-commerce price analysis. Existing internet data collection systems typically use pre-configured fixed parsing rules such as XPath, CSS selectors, or regular expressions to locate and extract target fields such as title, body text, author, and publication time from the document object model tree of the target webpage, and then store the collected structured data in a big data storage system for use by downstream models.

[0003] In existing technologies, because target websites periodically undergo page redesigns or structural adjustments, the original fixed parsing rules are prone to becoming invalid. When the location path of a certain field changes while other fields can still be extracted, the system often marks the record as "partially valid" and continues to flow into downstream processing nodes. This approach has the following problems: First, the extracted content may have semantic misalignment, such as mistakenly extracting the username in the navigation bar as the "author" field. Such errors are difficult to be detected by traditional missing data detection mechanisms when the field is not empty.

[0004] Second, semantically misaligned data entering downstream models can pollute the model's mapping relationships through the shared parameter system, causing the model to produce systematic biases for subsequent normal records. Third, once semantic pollution is detected, it is necessary to roll back massive amounts of historical data and perform full re-parsing and model retraining, which is extremely costly and cannot achieve targeted incremental repair.

[0005] Therefore, how to avoid the generation of semantically misaligned data, block its downstream propagation, and repair the affected models in a lightweight manner after contamination occurs are the technical challenges that urgently need to be solved in the field of Internet data collection. Summary of the Invention

[0006] This application provides an internet data collection method and system based on data analysis, which solves the problems in the prior art where semantic misalignment of fields caused by page structure evolution is difficult to identify, misaligned data pollutes downstream models and cannot be incrementally repaired. It achieves the technical effects of real-time verification and extraction of content based on the inherent semantic constraints of data, isolation of abnormal records to block the spread of pollution, and lightweight incremental correction of downstream models through counterfactual compensation data.

[0007] This application provides an Internet data collection method based on data analysis, including: acquiring historical verification data and constructing a semantic constraint representation of the target data field;

[0008] According to the initial collection rules, extract the current data fields from the current webpage, perform real-time feature comparison, and output the feature comparison results;

[0009] Based on the feature comparison results, hierarchical isolation processing is performed on the current data field. When the feature comparison result is unsuccessful, a backup feature search is performed on the current webpage. When the backup feature search fails, the current data record containing the current data field is stored in the abnormal isolation area.

[0010] Monitor the data increase rate within the abnormal isolation zone. When the data increase rate reaches the trigger condition, perform unsupervised candidate data cluster partitioning from the current webpage and compare the semantic center representation with the candidate data clusters to update the initial collection rules.

[0011] Identify contaminated data records, generate counterfactual compensation data, and use the counterfactual compensation data to perform incremental parameter compensation processing on downstream processing nodes that receive the current data records.

[0012] Furthermore, the steps for constructing the semantic constraint representation of the target data field include:

[0013] Fit the probability distributions of data length features and character category ratio features of a single data field in historical validation data to generate a single-field feature distribution representation;

[0014] Map a single data field in the historical verification data to a fixed-length vector, and calculate the center point of the fixed-length vector to generate a semantic center representation;

[0015] Calculate the numerical dependency features between different data fields in historical verification data to generate cross-field logical association representations.

[0016] Furthermore, the steps for outputting the feature comparison results include:

[0017] Calculate the current feature vector of the current data field and determine whether the current feature vector is within the feasible region represented by the single-field feature distribution.

[0018] Calculate the cosine similarity between the current data field and the semantic center representation, and determine whether the cosine similarity reaches the preset similarity boundary;

[0019] Based on other validation fields belonging to the current data record, check whether the current data field satisfies the cross-field logical association representation.

[0020] Furthermore, when the feature comparison result fails, the steps for searching for alternative features on the current webpage include:

[0021] When it is determined that some features of the current data field violate semantic constraints, candidate text blocks that meet the feature distribution range are matched from the current webpage's structure tree based on the single-field feature distribution representation.

[0022] Calculate the semantic distance between the candidate text block and the semantic center representation, and select the candidate text block with the smallest semantic distance as the backup feature.

[0023] Furthermore, the steps to store the current data record containing the current data field into the exception isolation area include:

[0024] Refuse to store the current data record in the mainstream database when no alternative feature can be matched;

[0025] Establish independent storage partitions for each data source within the abnormal isolation zone;

[0026] The current data record, the type of semantic constraint violation, the version number of the initial collection rule, and the snapshot path of the current webpage are all stored in a separate storage partition.

[0027] Furthermore, the step of performing unsupervised candidate data cluster partitioning from the current webpage when the data increment rate reaches the trigger condition includes:

[0028] Periodically calculate the percentage of newly added isolated records within an independent storage partition out of the total collected records that are considered to be in default.

[0029] Rule migration processing is initiated when the percentage of defaults exceeds the abnormal interception boundary.

[0030] Extract a target number of failed record page samples from independent storage partitions, and extract candidate elements from the failed record page samples that conform to the single-field feature distribution representation;

[0031] Candidate elements are aggregated into multiple independent data clusters based on text feature similarity.

[0032] Furthermore, the semantic center representation is compared with the candidate data clusters to update the initial collection rules:

[0033] Calculate the cluster-average semantic vector for each individual data cluster;

[0034] Compare the cluster average semantic vector with the semantic center representation;

[0035] The target data cluster with the smallest distance from the semantic center representation is selected as the new localization target. A new generation of extraction rules is generated based on the text density region of the target data cluster to replace the initial collection rules.

[0036] Further steps in identifying contamination data records and generating counterfactual compensation data include:

[0037] Locate semantically misaligned contaminated data records from the historical logs of the big data storage system;

[0038] Using the erroneous field content in the contaminated data record as the initial anchor point, a gradient bias step operation is performed in the direction of the legal domain prototype corresponding to the semantic constraint representation to generate alternative data samples that belong to the legal domain in terms of feature distribution as counterfactual compensation data.

[0039] Furthermore, the step of performing incremental parameter compensation processing on the downstream processing node that receives the current data record includes:

[0040] Pollution data records and counterfactual compensation data are paired and combined to form a feature correction training set;

[0041] The diagonal values ​​of the information matrix of the internal parameters of downstream processing nodes on the uncontaminated validation dataset are used as parameter importance weights.

[0042] Input the feature-corrected training set into the downstream processing node;

[0043] In the loss function for parameter fine-tuning, a penalty constraint term for importance weights is added to separately correct the parameter subspace corresponding to the contaminated data records in the downstream processing nodes.

[0044] This application provides an Internet big data acquisition system based on data analysis, which is used to implement an Internet data acquisition method based on data analysis, including: a semantic constraint representation construction module, a feature comparison result output module, a hierarchical isolation processing module, a data increase rate monitoring module, and an incremental parameter compensation module;

[0045] The semantic constraint representation construction module is used to acquire historical verification data and construct the semantic constraint representation of the target data field.

[0046] The feature comparison result output module is used to extract the current data field from the current webpage according to the initial collection rules, perform real-time feature comparison, and output the feature comparison result;

[0047] The hierarchical isolation processing module is used to perform hierarchical isolation processing on the current data field based on the feature comparison result. When the feature comparison result is unsuccessful, a backup feature search is performed on the current webpage. When the backup feature search fails, the current data record containing the current data field is stored in the abnormal isolation area.

[0048] The data increase rate monitoring module is used to monitor the data increase rate in the abnormal isolation zone. When the data increase rate reaches the trigger condition, it performs unsupervised candidate data cluster division from the current webpage and compares the semantic center representation with the candidate data cluster to update the initial collection rules.

[0049] The incremental parameter compensation module is used to identify contaminated data records, generate counterfactual compensation data, and use the counterfactual compensation data to perform incremental parameter compensation processing on the downstream processing node that receives the current data record.

[0050] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0051] By constructing semantic constraint representations of target data fields, the collection rules shift from relying on webpage structure to the intrinsic characteristics of the data. When a page redesign causes the original positioning rules to become invalid, the system not only checks whether a field is empty, but also compares its statistical characteristics, semantic attributes, and cross-field logical relationships in real time. This allows it to identify and intercept semantically misaligned content, preventing erroneous information such as usernames in the navigation bar from being misjudged as valid fields and improving the semantic accuracy of data collection.

[0052] Furthermore, this method performs hierarchical isolation processing on data that fails validation. After a backup feature search fails, the abnormal records are stored in an independent isolation area, preventing them from entering the mainstream database and downstream models. This cuts off the spread of semantically misaligned data to downstream knowledge graphs, classification, or recommendation systems, prevents erroneous mapping relationships from spreading in shared model parameters, and ensures the stability and reliability of big data analysis results.

[0053] Furthermore, for historical contaminated data already stored in the big data storage system, incremental parameter compensation is performed on downstream processing nodes by generating counterfactual compensation data, eliminating the need to roll back massive amounts of historical data or retrain the model entirely. Guided only by contaminated records and semantic constraints, the contaminated parameter subspace in the model is corrected in a targeted manner, compressing the traditional large-scale re-cleaning that takes several days into a lightweight incremental adjustment, reducing operation and maintenance costs and improving the system's long-term adaptability to site evolution. Attached Figure Description

[0054] Figure 1 A flowchart of an Internet data collection method based on data analysis provided in this application embodiment;

[0055] Figure 2 This is a schematic diagram of the structure of an Internet data acquisition system based on data analysis provided in an embodiment of this application. Detailed Implementation

[0056] This application provides an internet data collection method and system based on data analysis, which solves the problem in the prior art that semantic misaligned data caused by page structure evolution is difficult to identify, pollutes downstream models, and cannot be incrementally repaired. By constructing semantic constraint representations for real-time feature comparison and isolating abnormal records, and then using counterfactual compensation data to perform incremental parameter compensation on downstream processing nodes, the technical effect of accurately intercepting semantic misaligned data, blocking the spread of pollution, and performing lightweight targeted repair on polluted models is achieved.

[0057] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0058] like Figure 1 The diagram shown is a flowchart of an Internet data collection method based on data analysis provided in this application embodiment. The method is applied to an Internet data collection system based on data analysis and includes the following steps: acquiring historical verification data, and constructing a semantic constraint representation of the target data field based on the historical verification data. The semantic constraint representation includes a single-field feature distribution representation, a semantic center representation, and a cross-field logical association representation.

[0059] According to the initial collection rules, extract the current data field from the current webpage, and perform real-time feature comparison between the current data field and the single-field feature distribution representation, semantic center representation, and cross-field logical association representation, and output the feature comparison results.

[0060] Based on the feature comparison results, hierarchical isolation processing is performed on the current data field. When the feature comparison result is unsuccessful, a backup feature search is performed on the current webpage. When the backup feature search fails, the current data record containing the current data field is stored in the abnormal isolation area.

[0061] Monitor the data increase rate within the abnormal isolation zone. When the data increase rate reaches the trigger condition, perform unsupervised candidate data cluster partitioning from the current webpage and compare the semantic center representation with the candidate data clusters to update the initial collection rules.

[0062] Identify contaminated data records already stored in the big data storage system, generate counterfactual compensation data based on the contaminated data records and semantic constraint representations, and use the counterfactual compensation data to perform incremental parameter compensation processing on the downstream processing nodes that receive the current data records.

[0063] In this embodiment, within the overall processing framework of the Internet big data collection method based on data analysis, the system runs on a distributed computing system that includes a crawler server, a streaming feature extraction cluster, and a deep learning parameter server.

[0064] The process of acquiring historical verification data and constructing semantic constraint representations of target data fields based on this historical verification data involves retrieving high-confidence collection logs from the past six months containing manually verified webpage text, authors, and publication times as historical verification data.

[0065] An independent space is allocated in the memory of the streaming feature extraction cluster to quantify the structural and semantic features of the target data field, generating a semantic constraint representation file that can be serialized and stored. The current data field is extracted from the current webpage according to the initial acquisition rules. The current data field is then compared in real-time with single-field feature distribution representations, semantic center representations, and cross-field logical association representations, and the comparison results are output. This comparison process does not depend on whether the document object model tree hierarchy of the webpage has changed; instead, it directly reads the inherent attribute features of the current data field, forming a physically-based data interception network.

[0066] Based on the feature comparison results, hierarchical isolation processing is performed on the current data field. If the feature comparison result fails, a backup feature search is performed on the current webpage. If the backup feature search fails, the current data record containing the current data field is stored in the anomaly isolation zone. This anomaly isolation zone is located in a disk sector of the distributed file system that is physically isolated from the mainstream business database, preventing erroneous data from contaminating the normal business process.

[0067] Monitor the data growth rate within the abnormal isolation zone. When the data growth rate reaches the trigger condition, perform unsupervised candidate data cluster partitioning from the current webpage and compare the semantic center representation with the candidate data clusters to update the initial collection rules.

[0068] This process enables the system to self-heal when facing code refactoring of a target website. It identifies contaminated data records already stored in the big data storage system, generates counterfactual compensation data based on these records and semantic constraints, and uses this counterfactual compensation data to perform incremental parameter compensation processing on downstream processing nodes receiving the current data records. These downstream processing nodes include convolutional neural networks for text sentiment classification or converter encoder layers for knowledge graph construction. The incremental parameter compensation processing accurately eliminates network weight biases caused by semantic misalignment without requiring full retraining using all the massive amounts of historical validation data.

[0069] Furthermore, the steps for constructing the semantic constraint representation of the target data field include:

[0070] Fit the probability distributions of data length features and character category ratio features of a single data field in historical validation data to generate a single-field feature distribution representation;

[0071] Map a single data field in the historical verification data to a fixed-length vector, and calculate the center point of the fixed-length vector to generate a semantic center representation;

[0072] Calculate the numerical dependency features between different data fields in historical verification data to generate cross-field logical association representations.

[0073] In this embodiment, the specific implementation process for generating a single-field feature distribution representation by fitting the probability distributions of the data length feature and character category ratio feature of a single data field in the historical verification data is as follows: extract the character length value and the percentage of Chinese characters in a single data field from the historical verification data, and use a Gaussian mixture model to fit the feasible region boundary of its feature space. The formula for fitting the probability density value is:

[0074] ;

[0075] In this formula, Represents the probability density value. This refers to the extracted field feature values, specifically character length or percentage values ​​in this case. The total number of Gaussian distribution components is determined by the optimal silhouette coefficient for clustering. Indicates the component index of the Gaussian distribution. Indicates component weights, Represents the normal distribution function. Represents the mean of the components. This represents the standard deviation of the components.

[0076] Individual data fields in historical validation data are mapped to fixed-length vectors, and the center points of these fixed-length vectors are calculated to generate semantic center representations. A pre-trained lightweight language model is then used to convert the text into semantically dense vectors. The formula for generating semantic center representation vectors is as follows:

[0077] ;

[0078] in, Represents the semantic center representation vector. This represents the total number of historical verification data samples. This indicates the sample number of the historical verification data. This represents the semantic vector of historical samples. It calculates the numerical dependency features between different data fields in the historical validation data, generating cross-field logical association representations. Specifically, it establishes an association table to record numerical logical rules such as "the length of the main text data must be greater than five times the length of the title data" on the same page.

[0079] Furthermore, the steps for outputting the feature comparison results include:

[0080] Calculate the current feature vector of the current data field and determine whether the current feature vector is within the feasible region represented by the single-field feature distribution.

[0081] Calculate the cosine similarity between the current data field and the semantic center representation, and determine whether the cosine similarity reaches the preset similarity boundary;

[0082] Based on other validation fields belonging to the current data record, check whether the current data field satisfies the cross-field logical association representation.

[0083] In this embodiment, the current feature vector of the current data field is calculated, and it is determined whether the current feature vector lies within the feasible region of the single-field feature distribution representation. Specifically, the actual length and proportion of Chinese characters in the current data field are extracted to form the current feature vector, which is then substituted into the probability density formula. When the output probability density value is less than a preset confidence lower limit, it is determined that the data does not fall within the feasible region. The cosine similarity between the current data field and the semantic center representation is calculated, and it is determined whether the cosine similarity reaches a preset similarity boundary. The formula for calculating the cosine similarity value is:

[0084] ;

[0085] Wherein, Sim represents the cosine similarity score, with a value ranging from negative one to positive one. This represents the current feature vector generated by mapping the current data field using the same language model. Let |u| and |C| represent the semantic center representation vectors, where |u| and |C| represent the magnitudes of the two vectors, respectively.

[0086] When the cosine similarity value is less than the preset similarity boundary of 0.7, the feature comparison result is recorded as failing.

[0087] Based on other validated fields belonging to the same data record, check whether the current data field satisfies the cross-field logical association representation. For example, assuming the title field has passed the comparison, determine whether the length of the current body content violates the numerical dependency rule of being greater than five times the title length, thus forming a three-dimensional cross-feature validation system.

[0088] Furthermore, when the feature comparison result fails, the steps for searching for alternative features on the current webpage include:

[0089] When it is determined that some features of the current data field violate semantic constraints, candidate text blocks that meet the feature distribution range are matched from the current webpage's structure tree based on the single-field feature distribution representation.

[0090] Calculate the semantic distance between the candidate text block and the semantic center representation, and select the candidate text block with the smallest semantic distance as the backup feature.

[0091] In this embodiment, when it is determined that some features of the current data field violate semantic constraints, candidate text blocks that satisfy the feature distribution range are matched from the current webpage's structure tree based on the single-field feature distribution representation. The specific operation is as follows:

[0092] Instead of relying on preset tag paths, the crawler server traverses all the text nodes of the current webpage and extracts all text nodes whose character length is within the feasible range to form a candidate pool.

[0093] Calculate the semantic distance between the candidate text block and the semantic center representation, and select the candidate text block with the smallest semantic distance as the backup feature.

[0094] The computation unit is invoked to perform language model forward computation on each node in the candidate pool to obtain the feature vector. The formula for calculating the semantic distance is:

[0095] ;

[0096] in, Represents the semantic distance value. This represents the feature vector of the candidate text block. This represents the semantic center representation vector.

[0097] By iterating through and comparing all semantic distance values, the candidate text block with the smallest semantic distance value is locked, and its content is directly overwritten to replace the erroneous current data field extracted by the initial collection rules, thereby achieving online self-healing and repair of data without stopping the collection task.

[0098] Furthermore, the steps to store the current data record containing the current data field into the exception isolation area include:

[0099] Refuse to store the current data record in the mainstream database when no alternative feature can be matched;

[0100] Establish independent storage partitions for each data source within the abnormal isolation zone;

[0101] The current data record, the type of semantic constraint violation, the version number of the initial collection rule, and the snapshot path of the current webpage are all stored in a separate storage partition.

[0102] In this embodiment, the current data record is refused to be stored in the mainstream database when no alternative feature can be matched. The specific process is as follows:

[0103] When the backup feature search has traversed all candidate text blocks and still has not found any content with a semantic distance value that meets the standard, the write connection of the current data record to mainstream big data storage systems such as Elasticsearch or MySQL is cut off.

[0104] Within the anomaly isolation zone, independent storage partitions are established for each data source. The anomaly isolation zone is a specially designated cold storage area in the distributed file system. Independent folders are created according to the top-level domain name of the website being collected, ensuring that misaligned data from different sources are not mixed together.

[0105] The current data record, the type of semantic constraint violation, the version number of the initial collection rule, and the snapshot path of the current webpage are all stored in a separate storage partition.

[0106] The current original webpage code file is packaged and stored in a distributed file system. The current collection task number and the specific reason for violating the distributed representation or similarity representation are appended to the end of the current data record as metadata in the form of key-value pairs, forming a complete isolation audit report. This ensures that sufficient on-site environmental data is available for subsequent manual tracing or automatic machine analysis.

[0107] Furthermore, the step of performing unsupervised candidate data cluster partitioning from the current webpage when the data increment rate reaches the trigger condition includes:

[0108] Periodically calculate the percentage of newly added isolated records within an independent storage partition out of the total collected records that are considered to be in default.

[0109] Rule migration processing is initiated when the percentage of defaults exceeds the abnormal interception boundary.

[0110] Extract a target number of failed record page samples from independent storage partitions, and extract candidate elements from the failed record page samples that conform to the single-field feature distribution representation;

[0111] Candidate elements are aggregated into multiple independent data clusters based on text feature similarity.

[0112] In this embodiment, the default rate of newly added isolated records within an independent storage partition relative to the total number of collected records is calculated periodically. The streaming feature extraction cluster incorporates a sliding time window counter, and the formula for calculating the default rate is as follows:

[0113] ;

[0114] in, This indicates the percentage of defaults. This indicates the number of new isolation records added to this independent storage partition within the past hour. This indicates the total number of data collection records initiated for this domain within the same time window. When the percentage of violations exceeds the abnormal interception boundary, rule migration processing will be initiated.

[0115] When the calculated default rate exceeds 30% for three consecutive time windows, the system determines that the target webpage structure has been permanently redesigned and automatically activates the rule migration module. A target number of invalid record page samples are extracted from the independent storage partition, and candidate elements conforming to the single-field feature distribution representation are extracted from these samples.

[0116] Candidate elements are aggregated into multiple independent data clusters based on text feature similarity. A clustering algorithm is applied, using the term frequency-inverse document frequency (TIF) and tag level depth as feature dimensions, to aggregate candidate elements with similar physical locations and text structures into different data clusters. The goal is to minimize the sum of squared errors within each cluster; the formula for calculating the objective function is:

[0117] ;

[0118] in, This represents the numerical value of the sum of squared errors within the cluster. This represents the total number of cluster centers. Indicates the current cluster center index. Indicates the division to the th A collection of elements in an independent data cluster. Indicates the index of a candidate element in the set. For feature dimension, Indicates the first The candidate element in the th... Dimensional feature values, Indicates the first The cluster center vector of each independent data cluster is in the th... The numerical value of the dimension.

[0119] Furthermore, the semantic center representation is compared with the candidate data clusters to update the initial collection rules:

[0120] Calculate the cluster-average semantic vector for each individual data cluster;

[0121] Compare the cluster average semantic vector with the semantic center representation;

[0122] The target data cluster with the smallest distance from the semantic center representation is selected as the new localization target. A new generation of extraction rules is generated based on the text density region of the target data cluster to replace the initial collection rules.

[0123] In this embodiment, the cluster-average semantic vector of each independent data cluster is calculated. The semantic representation vectors of all candidate elements within the same independent data cluster are weighted, summed, and averaged to obtain a vector feature representing the overall semantic tendency of the data cluster.

[0124] Compare the cluster average semantic vector with the semantic center representation. Use the semantic distance numerical calculation formula to measure the degree of deviation between the cluster average semantic vector and the semantic center representation established in the clean history stage.

[0125] The target data cluster with the smallest distance from the semantic center representation is selected as the new localization target. A new generation of extraction rules is generated based on the text density region of the target data cluster to replace the initial collection rules.

[0126] In practice, the common ancestor document object model node of the source target data cluster in the failure record page sample is used to calculate the text density as the ratio of the number of Chinese characters to the number of code tags under that ancestor node. The formula for calculating the text density value is as follows:

[0127] ;

[0128] in, This represents the calculated text density value, expressed in characters per tag. This represents the total length of the plain text in the webpage node containing the target data cluster. This indicates the total number of code tags contained within the webpage node.

[0129] Extract code tag attributes containing the highest text density, such as class names, identifiers, or hierarchical paths, automatically generate new node path parsing expressions, and automatically push them to the crawler server's configuration center to override the old initial collection rules.

[0130] Further steps in identifying contamination data records and generating counterfactual compensation data include:

[0131] Locate semantically misaligned contaminated data records from the historical logs of the big data storage system;

[0132] Using the erroneous field content in the contaminated data record as the initial anchor point, a gradient bias step operation is performed in the direction of the legal domain prototype corresponding to the semantic constraint representation to generate alternative data samples that belong to the legal domain in terms of feature distribution as counterfactual compensation data.

[0133] In this embodiment, semantically misaligned contaminated data records are located from the historical logs of the big data storage system. The system scans data batches written to the database before triggering the anomaly interception boundary, and locates and extracts the erroneous objects based on the original webpage Uniform Resource Locator (URL) of the records. Using the erroneous field content in the contaminated data record as the initial anchor point, a gradient bias stepping operation is performed towards the prototype of the legal domain corresponding to the semantic constraint representation, generating alternative data samples that belong to the legal domain in the feature distribution as counterfactual compensation data. This process is based on the gradient backpropagation algorithm of the latent space of the variational autoencoder, and the bias stepping formula for calculating the feature vector is:

[0134] ;

[0135] in, This represents the feature vector of counterfactual compensation data. This represents the feature vector of the pollution data record. This represents the bias step learning rate, which controls the size of the step in each gradient descent iteration. Represents the gradient operator; This represents the numerical value of the feature distance loss function, used to measure the gap between the current feature and the semantic constraint representation boundary; This represents the semantic center representation vector. The counterfactual compensation data generated through this operation has a directional effect of erasing error biases in the text feature space.

[0136] Furthermore, the step of performing incremental parameter compensation processing on the downstream processing node that receives the current data record includes:

[0137] Pollution data records and counterfactual compensation data are paired and combined to form a feature correction training set;

[0138] The diagonal values ​​of the information matrix of the internal parameters of downstream processing nodes on the uncontaminated validation dataset are used as parameter importance weights.

[0139] Input the feature-corrected training set into the downstream processing node;

[0140] In the loss function for parameter fine-tuning, a penalty constraint term for importance weight is added to maintain the original value of internal parameters with larger importance weights, and the parameter subspace corresponding to the contaminated data record in the downstream processing node is modified separately.

[0141] In this embodiment, contaminated data records and counterfactual compensation data are paired and combined to form a feature correction training set. Data with erroneous semantic features and legitimate alternative data generated through bias stepping are constructed as positive and negative sample pairs in a one-to-one ratio to form a dedicated mini-batch training set, which is then input into the downstream classification or prediction network.

[0142] The diagonal values ​​of the information matrix of the internal parameters of downstream processing nodes on the uncontaminated validation dataset are used as parameter importance weights. To prevent the incremental compensation process from damaging the original predictive ability of the downstream model, an elastic weight consolidation mechanism is adopted. The formula for calculating the parameter importance weights is as follows:

[0143] ;

[0144] in, This represents the parameter importance weight calculated for the p-th parameter. This represents the total number of samples in the clean validation dataset. This indicates the sample number in the clean validation dataset. Indicates the verification data label, This indicates that the data input is being validated. This represents the complete set of internal parameters for downstream processing nodes. This represents the p-th internal parameter in the downstream processing node. This represents partial derivative operations.

[0145] The feature-corrected training set is input into the downstream processing nodes. The deep learning model server receives the positive and negative sample pairs and extracts text features during forward propagation.

[0146] In the loss function for parameter fine-tuning, a penalty constraint term for importance weight is added to maintain the original value of internal parameters with larger importance weights, and the parameter subspace corresponding to the contaminated data record in the downstream processing node is modified separately.

[0147] This operation is achieved by modifying the total loss function during backpropagation. The formula for the modified incremental parameter compensation of the total loss function is as follows:

[0148] ;

[0149] The secret formula term adds a secondary penalty term on top of the original task optimization. This represents the value of the total loss function after incremental parameter compensation. This represents the numerical value of the basic task loss function; This represents the penalty constraint coefficient, used to control the strength of regularization. This indicates the importance weight of the parameters calculated above; This represents the p-th intrinsic parameter of the current iteration step; Let p represent the p-th internal parameter before contamination. By directing the energy of gradient updates to parameter neurons with lower weights that have absorbed the bias of contaminated data, the local error memory of downstream big data analysis models is repaired.

[0150] like Figure 2 The diagram shown is a structural schematic of an Internet big data acquisition system based on data analysis provided in this application embodiment. The Internet big data acquisition system based on data analysis provided in this application embodiment includes: a semantic constraint representation construction module, a feature comparison result output module, a hierarchical isolation processing module, a data increment rate monitoring module, and an incremental parameter compensation module.

[0151] The semantic constraint representation construction module is used to acquire historical verification data and construct the semantic constraint representation of the target data field.

[0152] The feature comparison result output module is used to extract the current data field from the current webpage according to the initial collection rules, perform real-time feature comparison, and output the feature comparison result;

[0153] The hierarchical isolation processing module is used to perform hierarchical isolation processing on the current data field based on the feature comparison result. When the feature comparison result is unsuccessful, a backup feature search is performed on the current webpage. When the backup feature search fails, the current data record containing the current data field is stored in the abnormal isolation area.

[0154] The data increase rate monitoring module is used to monitor the data increase rate in the abnormal isolation zone. When the data increase rate reaches the trigger condition, it performs unsupervised candidate data cluster division from the current webpage and compares the semantic center representation with the candidate data cluster to update the initial collection rules.

[0155] The incremental parameter compensation module is used to identify contaminated data records, generate counterfactual compensation data, and use the counterfactual compensation data to perform incremental parameter compensation processing on the downstream processing node that receives the current data record.

[0156] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0157] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0158] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0161] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data analysis-based method for collecting internet data, characterized in that: Includes the following steps: Obtain historical verification data and construct semantic constraint representations of target data fields; According to the initial collection rules, extract the current data fields from the current webpage, perform real-time feature comparison, and output the feature comparison results; Based on the feature comparison results, hierarchical isolation processing is performed on the current data field. When the feature comparison result is unsuccessful, a backup feature search is performed on the current webpage. When the backup feature search fails, the current data record containing the current data field is stored in the abnormal isolation area. Monitor the data increase rate within the abnormal isolation zone. When the data increase rate reaches the trigger condition, perform unsupervised candidate data cluster partitioning from the current webpage and compare the semantic center representation with the candidate data clusters to update the initial collection rules. Identify contaminated data records, generate counterfactual compensation data, and use the counterfactual compensation data to perform incremental parameter compensation processing on downstream processing nodes that receive the current data records.

2. The Internet data collection method based on data analysis as described in claim 1, characterized in that, The steps for constructing a semantic constraint representation of the target data field include: Fit the probability distributions of data length features and character category ratio features of a single data field in historical validation data to generate a single-field feature distribution representation; Map a single data field in the historical verification data to a fixed-length vector, and calculate the center point of the fixed-length vector to generate a semantic center representation; Calculate the numerical dependency features between different data fields in historical verification data to generate cross-field logical association representations.

3. The Internet data collection method based on data analysis as described in claim 1, characterized in that, The steps for outputting feature alignment results include: Calculate the current feature vector of the current data field and determine whether the current feature vector is within the feasible region represented by the single-field feature distribution. Calculate the cosine similarity between the current data field and the semantic center representation, and determine whether the cosine similarity reaches the preset similarity boundary; Based on other validation fields belonging to the current data record, check whether the current data field satisfies the cross-field logical association representation.

4. The Internet data collection method based on data analysis as described in claim 1, characterized in that, When the feature comparison result fails, the steps for searching for alternative features on the current webpage include: When it is determined that some features of the current data field violate semantic constraints, candidate text blocks that meet the feature distribution range are matched from the current webpage's structure tree based on the single-field feature distribution representation. Calculate the semantic distance between the candidate text block and the semantic center representation, and select the candidate text block with the smallest semantic distance as the backup feature.

5. The Internet data collection method based on data analysis as described in claim 1, characterized in that, The steps to store the current data record containing the current data field into the exception isolation area include: Refuse to store the current data record in the mainstream database when no alternative feature can be matched; Establish independent storage partitions for each data source within the abnormal isolation zone; The current data record, the type of semantic constraint violation, the version number of the initial collection rule, and the snapshot path of the current webpage are all stored in a separate storage partition.

6. The Internet data collection method based on data analysis as described in claim 1, characterized in that, The steps for performing unsupervised candidate data cluster partitioning from the current webpage when the data growth rate reaches the trigger condition include: Periodically calculate the percentage of newly added isolated records within an independent storage partition out of the total collected records that are considered to be in default. Rule migration processing is initiated when the percentage of defaults exceeds the abnormal interception boundary. Extract a target number of failed record page samples from independent storage partitions, and extract candidate elements from the failed record page samples that conform to the single-field feature distribution representation; Candidate elements are aggregated into multiple independent data clusters based on text feature similarity.

7. The Internet data collection method based on data analysis as described in claim 1, characterized in that, The semantic center representation is compared with the candidate data clusters to update the initial collection rules. Calculate the cluster-average semantic vector for each individual data cluster; Compare the cluster average semantic vector with the semantic center representation; The target data cluster with the smallest distance from the semantic center representation is selected as the new localization target. A new generation of extraction rules is generated based on the text density region of the target data cluster to replace the initial collection rules.

8. The Internet data collection method based on data analysis as described in claim 1, characterized in that, The steps for identifying pollution data records and generating counterfactual compensation data include: Locate semantically misaligned contaminated data records from the historical logs of the big data storage system; Using the erroneous field content in the contaminated data record as the initial anchor point, a gradient bias step operation is performed in the direction of the legal domain prototype corresponding to the semantic constraint representation to generate alternative data samples that belong to the legal domain in terms of feature distribution as counterfactual compensation data.

9. The Internet data collection method based on data analysis as described in claim 1, characterized in that, The steps for incremental parameter compensation processing of the downstream processing node that receives the current data record include: Pollution data records and counterfactual compensation data are paired and combined to form a feature correction training set; The diagonal values ​​of the information matrix of the internal parameters of downstream processing nodes on the uncontaminated validation dataset are used as parameter importance weights. Input the feature-corrected training set into the downstream processing node; In the loss function for parameter fine-tuning, a penalty constraint term for importance weights is added to separately correct the parameter subspace corresponding to the contaminated data records in the downstream processing nodes.

10. An Internet big data acquisition system based on data analysis, used to implement the Internet data acquisition method based on data analysis as described in any one of claims 1-9, characterized in that, include: The module includes a semantic constraint representation construction module, a feature comparison result output module, a hierarchical isolation processing module, a data increment rate monitoring module, and an incremental parameter compensation module. The semantic constraint representation construction module is used to acquire historical verification data and construct the semantic constraint representation of the target data field. The feature comparison result output module is used to extract the current data field from the current webpage according to the initial collection rules, perform real-time feature comparison, and output the feature comparison result; The hierarchical isolation processing module is used to perform hierarchical isolation processing on the current data field based on the feature comparison result. When the feature comparison result is unsuccessful, a backup feature search is performed on the current webpage. When the backup feature search fails, the current data record containing the current data field is stored in the abnormal isolation area. The data increase rate monitoring module is used to monitor the data increase rate in the abnormal isolation zone. When the data increase rate reaches the trigger condition, it performs unsupervised candidate data cluster division from the current webpage and compares the semantic center representation with the candidate data cluster to update the initial collection rules. The incremental parameter compensation module is used to identify contaminated data records, generate counterfactual compensation data, and use the counterfactual compensation data to perform incremental parameter compensation processing on the downstream processing node that receives the current data record.

Images