Internet big data extraction method

Abstract

The invention discloses an Internet big data extraction method, and relates to the technical field of big data, and the method comprises the following extraction steps: capturing target data from a website through a data collection module, and inputting the target data into a processing module; and cleaning the target data through a processing module to remove labels, advertisements and invalid characters, converting the text data and the picture data into structured data, and outputting a data set DA. According to the method, the data quality can be comprehensively evaluated, the extracted data is ensured to have relatively high credibility and accuracy, the influence of false, outdated and invalid information on subsequent analysis and decision is eliminated, the practicability and value of the data are improved, and through feature extraction and optimization, the accuracy of the data is improved. According to the method, deep mining and comprehensive extraction of data features can be realized, extraction of more accurate and comprehensive feature vectors is facilitated, and a richer data basis is provided for subsequent data analysis and mining.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method for extracting Internet big data. Background Art

[0002] In today's rapidly developing information age, Internet big data has become a key factor in promoting the development of various industries, especially in limited fields such as financial risk control, medical health, and online education. The precise extraction and analysis of big data has immeasurable value in improving business efficiency, optimizing user experience, and enhancing the scientific nature of decision-making. Since Internet big data contains a large amount of data and information, it is necessary to accurately extract useful data from Internet big data.

[0003] Currently, most existing Internet big data extraction methods use web crawler technology to capture relevant information from public websites and then store the data. However, in actual use, it may be interfered with by a large amount of expired information, phishing website information and useless information, resulting in low data accuracy and practicality. In addition, traditional methods may be relatively simple when performing feature extraction and do not fully consider the complexity and diversity of the data. As a result, the extracted feature vectors may not fully reflect the essential information of the data, limiting the in-depth mining and utilization of the data. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for extracting Internet big data, which solves the problems raised in the above background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a method for extracting Internet big data, comprising the following extraction steps:

[0006] Step 1: Capture target data from the website through the data acquisition module and input the target data into the processing module;

[0007] Step 2: The target data is cleaned through the processing module to remove labels, advertisements, and invalid characters, and the text data and image data are converted into structured data, outputting the dataset DA.

[0008] Step 3: Input the data set DA into the calculation module. The calculation module first calculates the credibility of each data in the data set DA and outputs the data credibility CVB (DA i ), and then filter out the data DB with high credibility according to the credibility threshold i , for DB i Optimize and output the optimized feature vector FGH(DB i ), and then output the optimized data DC i , through the optimized data DC iThe comprehensive evaluation value SDF(DC i ), according to the comprehensive evaluation value SDF(DC i ) Sort the data and select the data with the highest comprehensive evaluation value as the final extraction result;

[0009] Step 4: Store the final extraction results through the storage module to facilitate subsequent analysis and application.

[0010] Optionally, the calculation module includes: a data credibility assessment submodule, a feature extraction submodule and an assessment and analysis submodule.

[0011] Optionally, the calculation formula of the data credibility assessment submodule is as follows:

[0012]

[0013] in:

[0014] CVB(DA i ) refers to data credibility;

[0015] DA i Refers to the i-th data in the data set, n refers to the data DA i Total number of attributes, CAA j Refers to the weight of the j-th attribute, CAB j Refers to data DA i The j-th attribute value of CAB j，max Refers to the attribute CAB j The maximum value in the entire dataset, CAC j Refers to the basic credibility of the j-th attribute, TN refers to the current time, and TP refers to the data DA i timestamp, ...refers to data DA i The total credibility of Refers to the time decay function used to consider the temporal freshness of the data;

[0016] The processing process of the credibility submodule is as follows: the data set DA is input into the credibility submodule, and the i-th data DA in the data set DA is used to calculate the credibility of the data set DA. i , and the attribute DA it contains i The j-th attribute value CAB j , and based on the basic credibility CAC of the j-th attribute j Output data credibility CVB(DA i );

[0017] According to CVB (DA i ) value, filter out the data DB with high credibility i .

[0018] Optionally, the calculation formula of the feature extraction submodule is as follows:

[0019]

[0020] in:

[0021] FGH(DB i ) refers to the optimized feature vector;

[0022] DB i Refers to the CVB (DA i ) selected high-reliability data, FA j Refers to data DB i The j-th eigenvalue, CVB(DB i ) refers to data DB i Credibility, m refers to data DB i The total number of features, FAs refers to the data DB i The average value of all eigenvalues, FA j ×log(1+CVB(DB i )) refers to weighting each eigenvalue, with the weight being log(1+CVB(DB i )) to emphasize the importance of data with higher credibility in feature extraction, Refers to the standard deviation of the eigenvector, indicating the degree of dispersion of the eigenvalue, Refers to normalizing the confidence to smooth its impact on feature optimization;

[0023] The processing process of the feature extraction submodule is as follows: the high-credibility data DB i Input to the feature extraction submodule and based on the data DB i Credibility CVB(DB i ) and data DB i The j-th eigenvalue FA j Output optimized feature vector FGH(DB i );

[0024] The optimized data is DC i , that is, DB i Data after optimization by the feature extraction submodule.

[0025] Optionally, the calculation formula of the evaluation and analysis submodule is as follows:

[0026]

[0027] in:

[0028] SDF(DC i ) refers to the comprehensive evaluation value of the data;

[0029] DC i Refers to the i-th data after feature optimization, CVB (DC i ) refers to data DC i Credibility, SDQ j Refers to the standard deviation of the jth feature, used to normalize the feature value, FGH (DC i ) refers to data DC i Optimized feature vector, FFS j Refers to the jth eigenvalue in the target eigenvector;

[0030] The processing process of the evaluation and analysis submodule is as follows: i Input to the evaluation and analysis submodule, and correspondingly convert the data DC i Data credibility CVB (DC i ) and data DC i The optimized feature vector FGH(DC i ) is input to the evaluation and analysis submodule, and the comprehensive evaluation value SDF(DC i ).

[0031] Optionally, the data acquisition module uses web crawler technology to capture target data on the website, and the crawler software runs on a high-performance server to ensure that a large number of data requests can be processed efficiently, using a proxy IP pool and a delay mechanism to avoid causing excessive pressure on the target website and being blocked by the anti-crawler mechanism.

[0032] Optionally, the target data captured by the acquisition module are specifically text, pictures, videos and tables, so as to ensure that the collected data covers different fields and topics, and increase the diversity and comprehensiveness of the data.

[0033] Optionally, the processing module further performs deduplication and normalization on the data to ensure the uniqueness of each piece of data.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The present invention outputs data credibility through a data credibility evaluation submodule. This submodule can take into account the time factor of the data and impose appropriate penalties on outdated data, thereby improving the timeliness of the evaluation results. It can also screen out data subsets with higher credibility, effectively eliminate the interference of useless information such as expired information, and improve the accuracy and practicality of the data. This submodule can remove low-credibility data to reduce noise and interference, which helps to improve the efficiency and accuracy of subsequent processing. Because high-credibility data is more likely to contain valuable information, this submodule can significantly improve the accuracy of data screening and can automatically identify and exclude a large amount of useless and low-quality data such as expired information and phishing website information, thereby improving the overall accuracy and practicality of the data. This submodule effectively avoids the one-sidedness of single-dimensional evaluation. With the addition of new data and the update of old data, this submodule can dynamically adjust the data credibility to ensure the continuity and stability of data quality.

[0036] 2. The present invention outputs the optimized feature vector through the feature extraction submodule. This submodule can consider the discrete degree between the eigenvalues ​​and further optimize and adjust the feature vector. The calculation of this submodule can improve the discrimination and effectiveness of the feature vector, thereby providing more accurate information for subsequent comprehensive evaluation. This submodule can efficiently extract key features from a large amount of raw data and further refine these features through optimization algorithms to form a more compact and representative feature set, thereby improving the efficiency and accuracy of data processing and making subsequent data analysis and decision-making processes more reliable. The feature extraction and optimization process can adaptively adjust parameters and methods to adapt to different types of data sets and task requirements, showing strong flexibility and adaptability.

[0037] 3. The present invention outputs a comprehensive evaluation value of the data through the evaluation and analysis submodule. This submodule takes into account the credibility of the data and also conducts evaluation based on the similarity between the optimized eigenvector and the target eigenvector, making the evaluation result more comprehensive and accurate. By introducing the standard deviation for normalization, this submodule can eliminate the dimensional differences between different eigenvalues, making the evaluation result more fair and reasonable. This submodule combines the data credibility and the optimized eigenvector for final screening to obtain the data subset that best meets the application requirements to ensure the quality and accuracy of the final data subset, providing a reliable foundation for subsequent analysis, mining or application. This submodule comprehensively considers the credibility of the data and the results of feature extraction, which helps to make more informed decisions in the data extraction process, ensuring that the final data set is both accurate and practical. The comprehensive evaluation and decision-making process can take into account multiple dimensions and factors, and realize comprehensive optimization and screening of data to provide a solid foundation for subsequent data analysis and application.

[0038] Fourth, the present invention iterates the weight of the jth attribute in the data credibility assessment submodule based on the comprehensive evaluation value of the data, so that a continuous cycle and optimization feedback form is formed between multiple groups of submodules in this method. This iterative form can automatically adjust the weight parameter CAA according to the actual performance of the data. j , so that the weight is more in line with the actual situation of the data, thereby improving the accuracy of data credibility calculation, and gradually approaching the optimal solution through iteration, calculating a more accurate weight parameter CAA j The calculation results of data credibility are made more accurate and stable. By optimizing the weight parameters, the present invention can more accurately calculate the data credibility, thereby screening out higher-quality data to provide strong support for subsequent data analysis and processing. Through iteration, it can gradually approach the optimal solution, thereby reducing unnecessary computational overhead and improving the efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of the steps of this Internet big data extraction method;

[0040] Figure 2 This is a schematic diagram of the overall structure of this Internet big data extraction method;

[0041] Figure 3 This is a schematic diagram of the structure of the computing module of this Internet big data extraction method. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] This internet big data extraction method is different from existing big data extraction methods. Existing extraction methods often lack an effective data credibility assessment mechanism, resulting in the extracted data being of low quality and containing false information, which affects the accuracy of subsequent analysis and decision-making. In addition, existing methods are often too simple in feature extraction and do not fully consider the complexity and diversity of the data. As a result, the extracted feature vectors may not fully reflect the essential information of the data, which limits the in-depth mining and utilization of the data.

[0044] The module of this Internet big data extraction method can achieve a comprehensive assessment of data quality through data credibility assessment, ensuring that the extracted data has high credibility and accuracy. The calculation of data credibility can intuitively reflect the credibility level of the data, providing strong support for subsequent analysis and decision-making. The present invention can also achieve in-depth mining and comprehensive extraction of data features. The calculation of optimized feature vectors can more comprehensively reflect the essential information of the data, providing a richer data foundation for subsequent data analysis and mining.

[0045] Example 1: Please refer to Figures 1 to 3 , this implementation provides a method for extracting Internet big data, including the following extraction steps:

[0046] Step 1: Capture target data from the website through the data acquisition module and input the target data into the processing module;

[0047] Step 2: The target data is cleaned through the processing module to remove labels, advertisements, and invalid characters, and the text data and image data are converted into structured data, outputting the dataset DA.

[0048] Step 3: Input the data set DA into the calculation module. The calculation module first calculates the credibility of each data in the data set DA and outputs the data credibility CVB (DA i ), and then filter out the data DB with high credibility according to the credibility threshold i , for DB i Optimize and output the optimized feature vector FGH(DB i ), and then output the optimized data DC i , through the optimized data DC i The comprehensive evaluation value SDF(DC i ), according to the comprehensive evaluation value SDF(DC i ) Sort the data and select the data with the highest comprehensive evaluation value as the final extraction result;

[0049] Step 4: The final extraction results are stored in a storage module for subsequent analysis and application;

[0050] The calculation module includes: data credibility assessment submodule, feature extraction submodule and evaluation analysis submodule.

[0051] In this embodiment: In this method, by calculating the data credibility, CVB (DA i) can effectively filter out useless or low-quality data such as expired information and phishing website information, improve the accuracy and efficiency of data extraction, and provide strong support for subsequent data analysis and application with high credibility data, which helps to discover potential laws and patterns in the data and improve the effect of data-driven decision-making. Through feature extraction and optimization, FGH (DB i ) can reduce the complexity and cost of data processing, improve the efficiency and accuracy of data processing, and the optimized feature set is more compact and representative, which helps to enhance the expressiveness and discrimination of data and provide strong support for subsequent data analysis and decision-making processes. Based on the comprehensive evaluation results, SDF(DC i ) can intelligently select the optimal data items to ensure the high quality and practicality of the final data set. Comprehensive evaluation and intelligent decision-making can help improve the decision-making effect in the data extraction process and provide more reliable data support for subsequent data analysis and application. By introducing formulas such as data credibility evaluation, feature extraction and optimization, and comprehensive evaluation and decision-making, this method realizes the intelligence and automation of the data extraction process, reduces manual intervention and errors, and improves the efficiency and accuracy of data extraction. Compared with traditional web crawlers, this method can automatically identify and exclude useless or low-quality data, and improve the intelligence level of data extraction. The multiple sub-modules of this method can flexibly adjust the evaluation indicators, feature extraction algorithms and Decision rules achieve flexibility and scalability in data processing, help cope with the diversity and complexity of Internet big data, and improve the efficiency and effectiveness of data processing. Through comprehensive evaluation and intelligent decision-making, this method can ensure the high quality and practicality of the final data set, provide strong support for subsequent data analysis and decision-making processes, help improve the effectiveness of data-driven decision-making, and provide more reliable data support for the business development and innovation of enterprises. In the comprehensive evaluation and decision-making process, this method can consider the security and privacy protection requirements of data, ensure that sensitive information will not be leaked or user privacy will be violated during the data extraction and application process, help enhance users' trust and support for the extraction and application of Internet big data, and promote the development of data security and privacy protection;

[0052] More specifically: by calculating the data credibility evaluation formula, high-quality data subsets can be screened out, thereby effectively reducing the interference of useless information and improving the accuracy and efficiency of data extraction. Through the calculation of feature extraction and optimization, more accurate and comprehensive feature vectors can be extracted, providing a richer data foundation for subsequent data analysis and mining. The multiple groups of sub-modules of this method all take into account multiple aspects of the data, such as attributes, timestamps and eigenvalues, and can be flexibly adjusted and optimized according to actual needs. This makes this method more flexible and adaptable when processing data of different types and fields. Through comprehensive evaluation and decision-making, the data can be comprehensively and accurately evaluated and sorted or screened according to the evaluation value, which provides strong support for subsequent decision-making, making the decision more scientific, accurate and reasonable. The multiple groups of sub-modules of this method and their calculation results not only solve the problems and shortcomings of the existing technology, but also provide new ideas and methods for the innovation and development of Internet big data technology, which will help promote the application and development of Internet big data technology in various fields and provide strong support for the digital transformation and intelligent upgrading of society.

[0053] See also Figures 1 to 3 ,The processing process of the data credibility assessment submodule is as follows:

[0054]

[0055] in:

[0056] CVB(DA i ) refers to data credibility;

[0057] DA i Refers to the i-th data in the data set, n refers to the data DA i Total number of attributes, CAA j Refers to the weight of the j-th attribute, CAB j Refers to data DA i The j-th attribute value of CAB j，max Refers to the attribute CAB j The maximum value in the entire dataset, CAC j Refers to the basic credibility of the j-th attribute, TN refers to the current time, and TP refers to the data DA i timestamp, ...refers to data DA i The total credibility of Refers to the time decay function used to consider the temporal freshness of the data;

[0058] The processing process of the credibility submodule is as follows: the data set DA is input into the credibility submodule, and the i-th data DA in the data set DA is i, and the attribute DA it contains i The j-th attribute value CAB j , and based on the basic credibility CAC of the j-th attribute j Output data credibility CVB (DA i );

[0059] According to CVB (DA i ) value, filter out the data DB with high credibility i .

[0060] In this embodiment: the calculation of this submodule can remove low-credibility data, reduce noise and interference, and help improve the efficiency and accuracy of subsequent processing. Because high-credibility data is more likely to contain valuable information, this submodule can significantly improve the accuracy of data screening, and can automatically identify and exclude a large amount of useless or low-quality data such as expired information and phishing website information, thereby improving the overall accuracy and practicality of the data. Data credibility assessment is a dynamic process that can be continuously updated and adjusted according to newly collected data and information to ensure continuous improvement of data quality. This submodule effectively avoids the one-sidedness of single-dimensional assessment. As new data is collected, With the addition of new data and the updating of old data, this submodule can dynamically adjust the data credibility to ensure the continuity and stability of data quality. This submodule can adaptively adjust the evaluation indicators and weights according to different data sets and task requirements to adapt to the diversity and complexity of Internet big data. By real-time monitoring and updating data, this submodule can ensure the real-time and accuracy of data credibility, providing technical support for data extraction and decision-making. This submodule improves the accuracy and efficiency of data screening and effectively excludes useless or low-quality data. This submodule enhances the accuracy and reliability of the data set and provides strong support for subsequent data analysis and application.

[0061] More specifically: This submodule not only considers the credibility of a single attribute, but also integrates the credibility of multiple attributes through weighted summation, making the evaluation results more comprehensive and accurate. This submodule can consider the time factor of the data and impose appropriate penalties on outdated data, thereby improving the timeliness of the evaluation results. It can also filter out data subsets with higher credibility, thereby effectively eliminating the interference of useless information such as expired information and phishing website information, and improving the accuracy and practicality of the data. In the process of data extraction and processing, it can be based on CVB (DA i ) values ​​to sort or filter the data, thereby optimizing the data processing flow and improving processing efficiency.

[0062] See also Figures 1 to 3 , the feature extraction submodule processing process is as follows:

[0063]

[0064] in:

[0065] FGH(DB i ) refers to the optimized feature vector;

[0066] DB i Refers to the CVB (DA i ) selected high-reliability data, FA j Refers to data DB i The j-th eigenvalue, CVB(DB i ) refers to data DB i Credibility, m refers to data DB i The total number of features, FAs refers to the data DB i The average value of all eigenvalues, FA j ×log(1+CVB(DB i )) refers to weighting each eigenvalue, with the weight being log(1+CVB(DB i )) to emphasize the importance of data with higher credibility in feature extraction, Refers to the standard deviation of the eigenvector, indicating the degree of dispersion of the eigenvalue, Refers to normalizing the confidence to smooth its impact on feature optimization;

[0067] The processing process of the feature extraction submodule is as follows: the high-reliability data DB i Input to the feature extraction submodule and based on the data DB i Credibility CVB(DB i ) and data DB i The j-th eigenvalue FA j Output optimized feature vector FGH(DB i );

[0068] The optimized data is DC i , that is, DB i Data after optimization by the feature extraction submodule.

[0069] In this embodiment: the feature optimization process will make the features of some data points more prominent or more in line with certain standards, thereby indirectly affecting the subsequent screening. The calculation of this submodule can improve the discrimination and effectiveness of the feature vector, providing more accurate information for the subsequent comprehensive evaluation. This submodule can efficiently extract key features from a large amount of raw data and further refine these features through the optimization algorithm to form a more compact and representative feature set FGH (DB i), which greatly improves the efficiency and accuracy of data processing, making subsequent data analysis and decision-making processes more reliable. The feature extraction and optimization process can adaptively adjust parameters and methods to adapt to different types of data sets and task requirements, showing strong flexibility and adaptability. This submodule adopts advanced feature extraction algorithms, which can quickly and accurately extract key features from a large amount of raw data. Through optimization methods such as dimensionality reduction, denoising, and feature selection, this submodule can flexibly adjust algorithms and parameters according to different feature extraction requirements to achieve scalability and flexibility of feature extraction. This submodule optimizes the data processing process and reduces the complexity and cost of data processing. Under the action of this submodule, the expressive power and discrimination of data features are improved, which helps to discover potential laws and patterns in the data.

[0070] More specifically: This submodule introduces data credibility CVB (DB i ) as the weight, the eigenvalues ​​are weighted so that the optimized eigenvectors are more consistent with the actual situation of the data. This submodule can consider the discrete degree between the eigenvalues ​​and further optimize and adjust the eigenvectors. The optimized eigenvectors can more accurately describe the characteristic information of the data, providing a richer data foundation for subsequent data analysis and mining.

[0071] See also Figures 1 to 3 ,The processing process of the evaluation and analysis submodule is as follows:

[0072]

[0073] in:

[0074] SDF(DC i ) refers to the comprehensive evaluation value of the data;

[0075] DC i Refers to the i-th data after feature optimization, CVB (DC i ) refers to data DC i Credibility, SDQ j Refers to the standard deviation of the jth feature, used to normalize the feature value, FGH (DC i ) refers to data DC i Optimized feature vector, FFS j Refers to the jth eigenvalue in the target eigenvector;

[0076] The processing process of the evaluation and analysis submodule is as follows: i Input to the evaluation and analysis submodule, and correspondingly convert the data DC i Data credibility CVB (DC i ) and data DC iThe optimized feature vector FGH(DC i ) is input to the evaluation and analysis submodule, and the comprehensive evaluation value SDF(DC i ).

[0077] In this embodiment: this submodule combines the credibility of the data and the optimized feature vector to perform final screening to obtain the data subset that best meets the application requirements, so as to ensure the quality and accuracy of the final data subset and provide a reliable foundation for subsequent analysis, mining or application. This submodule comprehensively considers the credibility of the data, the feature extraction results and other possible evaluation indicators, which helps to make more informed decisions in the data extraction process and ensure that the final data set is both accurate and practical. The comprehensive evaluation and decision-making process can take into account multiple dimensions and factors to achieve comprehensive optimization and screening of data and provide a solid foundation for subsequent data analysis and application. This submodule comprehensively considers the credibility of the data, the feature extraction results and other evaluation indicators, and realizes comprehensive evaluation and optimization of the data through complex mathematical models and algorithms. Based on the comprehensive evaluation results, this submodule can intelligently select the optimal data items to ensure the high quality and practicality of the final data set. The calculation of this submodule can cope with the noise and abnormal data in the Internet big data and maintain the stability and reliability of the evaluation results. SDF (DC i ) It can flexibly adjust evaluation indicators and decision-making rules according to different evaluation needs and decision-making goals, realize the customizability of evaluation and decision-making, achieve comprehensive evaluation and optimization of data, ensure the high quality and practicality of the final data set, provide a scientific basis for decision-making in the data extraction process, and improve the efficiency and effectiveness of data extraction;

[0078] More specifically: This submodule not only considers the credibility of the data, but also combines the similarity between the optimized feature vector and the target feature vector for evaluation, making the evaluation result more comprehensive and accurate. j Normalization is performed. This submodule can eliminate the dimension differences between different eigenvalues, making the evaluation results more fair and reasonable. By calculating SDF (DC i ) can conduct a comprehensive evaluation of the data and sort or filter it according to the evaluation value, thus providing strong support for subsequent decision making. In the case of limited resources, SDF (DC i ) values ​​to prioritize data, thereby optimizing resource allocation and improving resource utilization efficiency.

[0079] Furthermore, based on SDF (DC i ) The weight of the j-th attribute in the data credibility assessment submodule CAA jPerform cyclic iterations to form a continuous loop and optimization feedback between multiple groups of submodules in this method. The processing process is as follows:

[0080] Step 1: CAA j，new =CAA j，old +α×SDF(DC i );

[0081] Step 2: Establish the loop termination condition;

[0082] Condition 1: |CAA j，new -CAA j，old |<CS;

[0083] Condition 2: The number of iterations is 100;

[0084] in:

[0085] CAA j，new Refers to the weight of the j-th attribute after iteration;

[0086] CAA j，old refers to the weight of the jth attribute before iteration, α refers to the learning rate, and CS refers to the convergence threshold 0.001;

[0087] In this embodiment: This iterative form can automatically adjust the weight parameter CAA according to the actual performance of the data j , so that the weights are more consistent with the actual situation of the data, thereby improving the calculation accuracy of the data credibility, and gradually approaching the optimal solution through iteration, that is, finding a set of weight parameters CAA j The calculation results of data credibility are more accurate and stable. Since the algorithm can adaptively adjust the weight parameters, it has a certain degree of robustness to data fluctuations and noise, and can better cope with complex data environments. By optimizing the weight parameters, the data credibility can be calculated more accurately, thereby screening out higher quality data, providing strong support for subsequent data analysis and processing. The algorithm can adaptively adjust the weight parameters according to different data characteristics and application scenarios, so it has stronger adaptability and flexibility. Through iteration, it can gradually approach the optimal solution, thereby reducing unnecessary computational overhead and improving the efficiency of the algorithm. By improving the accuracy and efficiency of data extraction, this iteration form can promote the development of Internet big data analysis, data mining and other fields, and provide more reliable data support for research and application in related fields. This iteration form provides new ideas and methods for algorithm innovation and optimization, which helps to promote technological innovation and progress in related fields. In practical applications, this iteration form can improve the effect of data extraction and analysis, and provide more accurate and reliable data support for corporate decision-making and business development.

[0088] In the specific implementation process, the multiple sub-modules in this method are used to form an Internet big data extraction system. By inputting the data set DA into the credibility sub-module, according to the i-th data DA in the data set DA, i , output data credibility CVB (DA i ), this submodule can remove low-credibility data to reduce noise and interference, which helps improve the efficiency and accuracy of subsequent processing. High-credibility data is more likely to contain valuable information. This submodule can significantly improve the accuracy of data screening and can automatically identify and exclude a large amount of useless or low-quality data such as expired information and phishing website information, thereby improving the overall accuracy and practicality of the data. This submodule effectively avoids the one-sidedness of single-dimensional evaluation. As new data is added and old data is updated, this submodule can dynamically adjust the data credibility to ensure the continuity and stability of data quality;

[0089] The highly reliable data DB i Input to the feature extraction submodule to output the optimized feature vector FGH(DB i ), in the process of feature optimization, the features of some data points will be more prominent or more in line with certain standards, which will indirectly affect the subsequent screening. This submodule calculation can improve the discrimination and effectiveness of the feature vector to provide more accurate information for the subsequent comprehensive evaluation. This submodule can efficiently extract key features from a large amount of raw data, and further refine these features through the optimization algorithm to form a more compact and representative feature set FGH (DB i ), which improves the efficiency and accuracy of data processing, making subsequent data analysis and decision-making processes more reliable. The feature extraction and optimization process can adaptively adjust parameters and methods to suit different types of data sets and task requirements, demonstrating strong flexibility and adaptability;

[0090] Data DC i The comprehensive evaluation value SDF(DC) of the output data of the evaluation analysis submodule is input i ), this submodule combines data credibility and optimized feature vectors for final screening, obtaining the data subset that best meets application requirements to ensure the quality and accuracy of the final data subset, providing a reliable foundation for subsequent analysis, mining, or application. This submodule comprehensively considers the credibility of the data and the results of feature extraction, which helps to make more informed decisions during the data extraction process, ensuring that the final data set is both accurate and practical. The comprehensive evaluation and decision-making process can take into account multiple dimensions and factors, achieve comprehensive optimization and screening of data, and provide a solid foundation for subsequent data analysis and application;

[0091] Based on SDF (DC i ) The weight of the j-th attribute in the data credibility assessment submodule CAA jPerform cyclic iterations, so that multiple groups of submodules in this method form a continuous cycle and optimization feedback. This iterative form can automatically adjust the weight parameter CAA according to the actual performance of the data. j , so that the weights are more consistent with the actual situation of the data, thereby improving the accuracy of data credibility calculation, and gradually approaching the optimal solution through iteration, that is, calculating a more accurate weight parameter CAA j The calculation results of data credibility are made more accurate and stable. Since the algorithm can adaptively adjust the weight parameters, it has a certain robustness to data fluctuations and noise, and can better cope with complex data environments. By optimizing the weight parameters, the data credibility can be calculated more accurately, thereby screening out higher-quality data to provide strong support for subsequent data analysis and processing. Through iteration, it can gradually approach the optimal solution, thereby reducing unnecessary computing overhead and improving the efficiency of the algorithm. By improving the accuracy and efficiency of data extraction, this iterative form can promote the development of Internet big data analysis, data mining and other fields, and provide more reliable data support for research and applications in related fields.

[0092] This allows the various sub-modules to cooperate with each other in calculations, and to perform overall cycles and iterations, so that the overall system has the effect of automatic optimization and updating, and thus better adaptability.

[0093] Example 2: Please refer to Figure 1 、 Figure 2 and Figure 3 The data collection module uses web crawler technology to crawl target data on the website, and the crawler software runs on a high-performance server to ensure that it can efficiently process a large number of data requests. It uses a proxy IP pool and a delay mechanism to avoid excessive pressure on the target website and being blocked by the anti-crawler mechanism. The target data captured by the collection module are specifically text, pictures, videos and tables to ensure that the collected data covers different fields and topics, increasing the diversity and comprehensiveness of the data. The processing module also deduplicates and normalizes the data to ensure the uniqueness of each piece of data.

[0094] In this embodiment: the data acquisition module and the processing module are used to crawl target data from the Internet, upload and store it, and process and clean the data to facilitate subsequent calculations and analysis, thereby playing a basic role in subsequent calculations and analysis, and providing a large amount of data support for subsequent calculations and analysis.

[0095] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for extracting Internet big data, characterized by: The extraction steps include: Step 1: Capture target data from the website through the data acquisition module and input the target data into the processing module; Step 2: The target data is cleaned through the processing module to remove labels, advertisements, and invalid characters, and the text data and image data are converted into structured data, outputting the dataset DA. Step 3: Input the data set DA into the calculation module. The calculation module first calculates the credibility of each data in the data set DA and outputs the data credibility CVB (DA i ), and then filter out the data DB with high credibility according to the credibility threshold i , for DB i Optimize and output the optimized feature vector FGH(DB i ), and then output the optimized data DC i , through the optimized data DC i The comprehensive evaluation value SDF(DC i ), according to the comprehensive evaluation value SDF(DC i ) Sort the data and select the data with the highest comprehensive evaluation value as the final extraction result; Step 4: Store the final extraction results through the storage module to facilitate subsequent analysis and application.

2. The method for extracting Internet big data according to claim 1, wherein: The calculation module includes: a data credibility evaluation submodule, a feature extraction submodule and an evaluation and analysis submodule.

3. The method for extracting Internet big data according to claim 2, wherein: The calculation formula of the data credibility evaluation submodule is as follows: in: CVB(DA i ) refers to data credibility; DA i Refers to the i-th data in the data set, n refers to the data DA i Total number of attributes, CAA j Refers to the weight of the j-th attribute, CAB j Refers to data DA i The j-th attribute value of CAB j，max Refers to the attribute CAB j The maximum value in the entire dataset, CAC j Refers to the basic credibility of the j-th attribute, TN refers to the current time, and TP refers to the data DA i timestamp, Refers to data DA i The total credibility of Refers to the time decay function used to consider the temporal freshness of the data; The processing process of the credibility submodule is as follows: the data set DA is input into the credibility submodule, and the i-th data DA in the data set DA is used to calculate the credibility of the data set DA. i , and the attribute DA it contains i The j-th attribute value CAB j , and based on the basic credibility CAC of the j-th attribute j Output data credibility CVB(DA i ); According to CVB (DA i ) value, filter out the data DB with high credibility i .

4. The method for extracting Internet big data according to claim 3, wherein: The calculation formula of the feature extraction submodule is as follows: in: FGH(DB i ) refers to the optimized feature vector; DB i Refers to the CVB (DA i ) selected from the highly credible data, FA j Refers to data DB i The j-th eigenvalue, CVB(DB i ) refers to data DB i Credibility, m refers to data DB i The total number of features, FAs refers to the data DB i The mean of all eigenvalues, FA j ×log(1+CVB(DB i )) refers to weighting each eigenvalue, with the weight being log(1+CVB(DB i )) to emphasize the importance of data with higher credibility in feature extraction, Refers to the standard deviation of the eigenvector, indicating the degree of dispersion of the eigenvalue, Refers to normalizing the confidence to smooth its impact on feature optimization; The processing process of the feature extraction submodule is as follows: the high-credibility data DB i Input to the feature extraction submodule and based on the data DB i Credibility CVB(DB i ) and data DB i The j-th eigenvalue FA j Output optimized feature vector FGH(DB i ); The optimized data is DC i , that is, DB i Data after optimization by the feature extraction submodule.

5. The method for extracting Internet big data according to claim 4, wherein: The calculation formula of the evaluation and analysis submodule is as follows: in: SDF(DC i ) refers to the comprehensive evaluation value of the data; DC i Refers to the i-th data after feature optimization, CVB (DC i ) refers to data DC i Credibility, SDQ j Refers to the standard deviation of the jth feature, used to normalize the feature value, FGH (DC i ) refers to data DC i Optimized feature vector, FFS j Refers to the jth eigenvalue in the target eigenvector; The processing process of the evaluation and analysis submodule is as follows: i Input to the evaluation and analysis submodule, and correspondingly convert the data DC i Data credibility CVB (DC i ) and data DC i The optimized feature vector FGH(DC i ) is input to the evaluation and analysis submodule, and the comprehensive evaluation value SDF(DC i ).

6. The method for extracting Internet big data according to claim 1, wherein: The data acquisition module uses web crawler technology to capture target data on the website, and the crawler software runs on a high-performance server to ensure that it can efficiently process a large number of data requests. It uses a proxy IP pool and a delay mechanism to avoid causing excessive pressure on the target website and being blocked by the anti-crawler mechanism.

7. The method for extracting Internet big data according to claim 6, characterized in that: The target data captured by the acquisition module are specifically text, pictures, videos and tables, so as to ensure that the collected data covers different fields and topics, and increase the diversity and comprehensiveness of the data.

8. The method for extracting Internet big data according to claim 1, wherein: The processing module also performs deduplication and normalization on the data to ensure the uniqueness of each piece of data.

Images