An internet big data extraction method

Abstract

The application discloses an internet big data extraction method, and relates to the technical field of big data, and comprises the following extraction steps: target data is captured from a website through a data acquisition module, and the target data is input into a processing module; the target data is cleaned by the processing module to remove labels, advertisements and invalid characters, and text data and picture data are converted into structured data, and a data set DA is output. The application can comprehensively evaluate data quality, ensure that the extracted data has high reliability and accuracy, help eliminate the influence of false, outdated and invalid information on subsequent analysis and decision-making, improve the practicality and value of data, and through feature extraction and optimization, the application can realize in-depth mining and comprehensive extraction of data features, help extract more accurate and comprehensive feature vectors, and provide a richer data basis for subsequent data analysis and mining.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to an internet big data extraction method. BACKGROUND

[0002] In today's highly developed information technology, internet big data has become a key factor in promoting the development of various industries, especially in the fields of financial risk control, medical health, online education, etc. The accurate extraction and analysis of big data have immeasurable value in improving business efficiency, optimizing user experience, and enhancing the scientific nature of decision-making. Since internet big data contains a large amount of data and information, it is necessary to accurately extract useful data from internet big data.

[0003] The existing internet big data extraction methods currently use web crawler technology to extract relevant information from public websites and then store the data. However, in actual use, they may be disturbed by a large amount of outdated information, phishing website information, and useless information, resulting in low data accuracy and practicality. Moreover, the traditional method may be relatively simple in feature extraction, without fully considering the complexity and diversity of data, which may limit the depth of data mining and utilization. SUMMARY

[0004] The present application aims to provide an internet big data extraction method that solves the problems mentioned in the background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solution: an internet big data extraction method, comprising the following extraction steps:

[0006] Step one, through the data acquisition module, target data is extracted from the website and input into the processing module;

[0007] Step two, the target data is cleaned by the processing module to remove labels, advertisements, and invalid characters, and the text data and picture data are converted into structured data, and the data set DA is output;

[0008] Step three, the data set DA is input into the calculation module, which first calculates the credibility of each data in the data set DA and outputs the data credibility CVB(DA i ), then filters out data DB i with high credibility according to the credibility threshold, i optimizes DB i to output the optimized feature vector FGH(DB i ), and then outputs the optimized data DC iThe overall evaluation value of the output data SDF (DC) i According to the comprehensive evaluation value SDF (DC) i Sort the data and select the data with the highest comprehensive evaluation value as the final extraction result;

[0009] Step 4: Store the final extracted results through the storage module for subsequent analysis and application.

[0010] Optionally, the calculation module includes: a data credibility assessment submodule, a feature extraction submodule, and an evaluation analysis submodule.

[0011] Optionally, the calculation formula for the data credibility assessment submodule is as follows:

[0012]

[0013] in:

[0014] CVB(DA i () refers to the credibility of the data;

[0015] DA i The i-th data point in the dataset refers to the data DA. i Total number of attributes, CAA j Refers to the weight of the j-th attribute, CAB j Refers to data DA i The j-th attribute value, CAB j，max Referencing attribute CAB j Maximum value in the entire dataset, CAC j The base credibility of the j-th attribute is referred to as TN, the current time is referred to as TP, and the data DA is referred to as TP. i timestamp, ...referring to data DA i Overall credibility The time decay function is used to consider the time freshness of data.

[0016] The processing procedure of the credibility submodule is as follows: Input the dataset DA into the credibility submodule, and process the data according to the i-th data DA in the dataset DA. i , and its included attribute DA i The j-th attribute value CAB j And based on the basic credibility CAC of the j-th attribute j Output data credibility CVB(DA) i );

[0017] According to CVB(DA) i The value of ) is used to filter out data with high credibility in the DB database. i .

[0018] Optionally, the calculation formula of the feature extraction submodule is as follows:

[0019]

[0020]

[0021] Wherein:

[0022] FGH(DB i ) refers to the optimized feature vector;

[0023] DB i refers to the data with high credibility filtered from the CVB(DA i ), FA j refers to the jth feature value of the data DB i , CVB(DB i ) refers to the credibility of the data DB i , m refers to the total number of features of the data DB i , FAs refers to the average value of all feature values of the data DB i , FA j ×log(1+CVB(DB i )) refers to weighting each feature value, and the weight is log(1+CVB(DB i )) to emphasize the importance of data with high credibility in feature extraction, refers to the standard deviation of the feature vector, indicating the dispersion degree of the feature value, refers to the normalization processing of the credibility to smooth its influence on feature optimization;

[0024] The processing process of the feature extraction submodule is as follows: input the data DB i with high credibility to the feature extraction submodule, and output the optimized feature vector FGH(DB i ) based on the credibility CVB(DB i ) of the data DB i and the jth feature value FA j of the data DB i ;

[0025] The optimized data is DC i , that is, the data DB i optimized by the feature extraction submodule.

[0026] Optionally, the calculation formula of the evaluation analysis submodule is as follows:

[0027]

[0028] in:

[0029] SDF(DC i () refers to the comprehensive evaluation value of the data;

[0030] DC i Refers to the i-th data after feature optimization, CVB(DC) i (Refers to data DC) i Credibility, SDQ j Refers to the standard deviation of the j-th feature, used to normalize the eigenvalues, FGH(DC) i (Refers to data DC) i Optimized feature vectors, FFS j Refers to the j-th eigenvalue in the target eigenvector;

[0031] The processing procedure of the evaluation and analysis submodule is as follows: Data DC i Input to the evaluation and analysis submodule, and correspondingly convert the data DC. i Data credibility CVB (DC) i ) and data DC i Optimized feature vector FGH(DC) i The data is input into the evaluation and analysis submodule, and the output data is the comprehensive evaluation value SDF (DC). i ).

[0032] Optionally, the data acquisition module uses web crawler technology to crawl target data on the website, and the crawler software runs on a high-performance server to ensure that it can efficiently handle a large number of data requests. Proxy IP pools and delay mechanisms are used to avoid putting too much pressure on the target website and being blocked by anti-crawler mechanisms.

[0033] Optionally, the target data captured by the acquisition module may specifically be text, images, videos, and tables, to ensure that the acquired data covers different fields and topics, increasing the diversity and comprehensiveness of the data.

[0034] Optionally, the processing module also performs deduplication and normalization on the data to ensure the uniqueness of each data entry.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] The data credibility evaluation submodule of the present application outputs data credibility, which can consider the time factor of data to appropriately punish outdated data, thereby improving the timeliness of the evaluation result, and can also filter out a data subset with higher credibility, effectively excluding the interference of expired information and other useless information, improving the accuracy and practicality of the data, removing low-credibility data to reduce noise and interference, which helps to improve the efficiency and accuracy of subsequent processing, and high-credibility data is more likely to contain valuable information, so that the submodule can significantly improve the accuracy of data filtering, automatically identify and exclude a large amount of expired information, phishing website information and other useless and low-quality data, and improve the overall accuracy and practicality of the data, the submodule effectively avoids one-sidedness in single-dimensional evaluation, and can dynamically adjust the data credibility with the addition of new data and the update of old data, ensuring the continuity and stability of data quality.

[0037] The feature extraction submodule of the present application outputs an optimized feature vector, which can further optimize and adjust the feature vector by considering the discrete degree between feature values, improve the discrimination and effectiveness of the feature vector, and provide more accurate information for subsequent comprehensive evaluation, and the submodule can efficiently extract key features from a large amount of raw data and further refine these features through an optimization algorithm to form a more compact and representative feature set, thereby improving the efficiency and accuracy of data processing and making the subsequent data analysis and decision-making process more reliable.

[0038] The evaluation and analysis submodule of the present application outputs a comprehensive evaluation value of the data, which considers the credibility of the data and evaluates the similarity between the optimized feature vector and the target feature vector, making the evaluation result more comprehensive and accurate, and by introducing standard deviation for normalization processing, the submodule can eliminate the dimensional difference between different feature values, making the evaluation result more fair and reasonable, and the submodule combines data credibility and the optimized feature vector for final screening to obtain a data subset that best meets application requirements, thereby ensuring the quality and accuracy of the final data subset and providing a reliable basis for subsequent analysis, mining or application, and the submodule comprehensively considers the credibility of the data and the results of feature extraction, which helps to make more intelligent decisions during data extraction, ensuring that the final data set is accurate and practical, and the comprehensive evaluation and decision-making process can consider multiple dimensions and factors to achieve comprehensive optimization and screening of the data, thereby providing a solid foundation for subsequent data analysis and application.

[0039] Fourthly, the weight of the jth attribute in the data credibility evaluation submodule is cyclically iterated based on the comprehensive evaluation value of the data, so that the multiple sets of submodules in the method form a form of continuous cycle and optimization feedback, and the iteration form can automatically adjust the weight parameter CAA according to the actual performance of the data j , so that the weight is more in line with the actual situation of the data, thereby improving the accuracy of data credibility calculation, and the optimal solution can be gradually approached through the iteration form, and a more accurate weight parameter CAA is calculated j The calculation result of the data credibility is more accurate and stable, the data credibility can be more accurately calculated by optimizing the weight parameter, so that higher quality data is screened out to provide strong support for subsequent data analysis and processing, and the optimal solution can be gradually approached through the iteration form, so as to reduce unnecessary calculation overhead and improve the efficiency of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The step flowchart of the present internet big data extraction method is shown in the figure.

[0041] Figure 2 The overall structure schematic diagram of the present internet big data extraction method is shown in the figure.

[0042] Figure 3 The structure schematic diagram of the calculation module of the present internet big data extraction method is shown in the figure. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] Regarding the present internet big data extraction method, unlike the existing big data extraction method, the existing extraction method often lacks an effective data credibility evaluation mechanism, resulting in the extracted data may have problems such as low quality, false information, etc., affecting the accuracy of subsequent analysis and decision-making, and the existing method is often too simple in feature extraction, without fully considering the complexity and diversity of data, resulting in the extracted feature vector may not fully reflect the essential information of the data, limiting the depth mining and utilization of data.

[0045] And the module of the internet big data extraction method can realize comprehensive evaluation of data quality through data credibility evaluation, ensure that the extracted data has high credibility and accuracy, the calculation of data credibility can intuitively reflect the credibility level of the data, provide strong support for subsequent analysis and decision-making, and the application can realize in-depth mining and comprehensive extraction of data characteristics, the calculation of the optimized feature vector can more comprehensively reflect the essential information of the data, and provide a more abundant data basis for subsequent data analysis and mining.

[0046] Embodiment one: please refer to Figures 1 to 3 The embodiment provides an internet big data extraction method, which comprises the following extraction steps:

[0047] Step one, target data is grabbed from a website through a data acquisition module, and the target data is input into a processing module;

[0048] Step two, the target data is cleaned by the processing module to remove labels, advertisements and invalid characters, and text data and picture data are converted into structured data, and a data set DA is output;

[0049] Step three, the data set DA is input into a calculation module, the calculation module first calculates the credibility of each piece of data of the data set DA, and outputs data credibility CVB(DA i ), then according to the credibility threshold, data DB i with high credibility is screened out i DB i is optimized to output an optimized feature vector FGH(DB i ), then an optimized data DC i is output, a comprehensive evaluation value SDF(DC i ) of the data is output through the optimized data DC i , the data is sorted according to the comprehensive evaluation value SDF(DC i ), and data with a high comprehensive evaluation value is selected as the final extraction result;

[0050] Step four, the final extraction result is stored through a storage module, so as to facilitate subsequent analysis and application;

[0051] The calculation module comprises a data credibility evaluation submodule, a feature extraction submodule and an evaluation analysis submodule.

[0052] In the embodiment, the data credibility CVB(DA i) can effectively filter out useless or low-quality data such as expired information and phishing website information, improve the accuracy and efficiency of data extraction, and provide strong support for subsequent data analysis and application, which helps to discover potential rules and patterns in data, improve data-driven decision-making effect, through feature extraction and optimization, FGH(DB i ) can reduce the complexity and cost of data processing, improve the efficiency and accuracy of data processing, and the optimized feature set is more compact and representative, which helps to enhance the expression ability and discrimination of data, and provides strong support for subsequent data analysis and decision-making process, based on the comprehensive evaluation result, SDF(DC i ) can intelligently select the optimal data item to ensure the high quality and practicality of the final data set, and the comprehensive evaluation and intelligent decision-making help to improve the decision-making effect in the data extraction process and provide more reliable data support for subsequent data analysis and application, by introducing data credibility evaluation, feature extraction and optimization, and comprehensive evaluation and decision-making formulas, this method realizes the intelligence and automation of the data extraction process, reduces manual intervention and errors, improves the efficiency and accuracy of data extraction, compared with the traditional web crawler method, this method can automatically identify and exclude useless or low-quality data, improving the intelligent level of data extraction, the multiple sub-modules of this method can flexibly adjust the evaluation index, feature extraction algorithm and decision-making rule according to different data sets and task requirements, realizing the flexibility and scalability of data processing, which helps to cope with the diversity and complexity of Internet big data, improves the efficiency and effect of data processing, through comprehensive evaluation and intelligent decision-making, this method can ensure the high quality and practicality of the final data set, providing strong support for subsequent data analysis and decision-making process, which helps to improve the data-driven decision-making effect, providing more reliable data support for the business development and innovation of enterprises, in the comprehensive evaluation and decision-making process, this method can consider the security and privacy protection requirements of data, ensuring that sensitive information is not leaked or user privacy is not violated during data extraction and application, which helps to enhance the trust and support of users for Internet big data extraction and application, and promotes the development of data security and privacy protection;

[0053] More specifically: through the calculation of the data credibility evaluation formula, a high-quality data subset can be screened out, thereby effectively reducing the interference of useless information and improving the accuracy and efficiency of data extraction. Through the calculation of feature extraction and optimization, more accurate and comprehensive feature vectors can be extracted, providing a more abundant data basis for subsequent data analysis and mining. The multiple sub-modules of the method consider multiple aspects of data, such as attributes, timestamps, and feature values, and can be flexibly adjusted and optimized according to actual needs, which makes the method more flexible and adaptable when dealing with different types and fields of data. Through comprehensive evaluation and decision-making, data can be evaluated comprehensively and accurately, and sorted or screened according to the evaluation value, which provides strong support for subsequent decision-making, making the decision-making more scientific, accurate and reasonable. The multiple sub-modules of the method and their calculation results not only solve the problems and deficiencies in the prior art, but also provide new ideas and methods for the innovation and development of internet big data technology, which helps to promote the application and development of internet big data technology in various fields and provides strong support for the digital transformation and intelligent upgrading of society.

[0054] Please refer to Figures 1 to 3 , the processing process of the data credibility evaluation sub-module is as follows:

[0055]

[0056] Among them:

[0057] CVB(DA i ) refers to the data credibility;

[0058] DA i refers to the ith data in the data set, n refers to the total number of attributes of the data DA i , CAA j refers to the weight of the jth attribute, CAB j refers to the jth attribute value of the data DA i , CAB j，max refers to the maximum value of the attribute CAB j in the entire data set, CAC j refers to the basic credibility of the jth attribute, TN refers to the current time, TP refers to the timestamp of the data DA i , … refers to the total credibility of the data DA i , refers to a time decay function used to consider the time freshness of the data;

[0059] The processing process of the credibility sub-module is as follows: input the data set DA into the credibility sub-module, and according to the ith data DA iAnd the attributes DA contained therein i The jth attribute value of CAB j And based on the base credibility of the jth attribute CAC j Output data credibility CVB(DA i );

[0060] According to the value of CVB(DA i ), data DB with high credibility is screened out i .

[0061] In this embodiment: the calculation of the sub-module can remove low credibility data, reduce noise and interference, and help improve the efficiency and accuracy of subsequent processing. Because high credibility data is more likely to contain valuable information, the sub-module can significantly improve the accuracy of data screening, automatically identify and exclude a large amount of outdated information, phishing website information, and other useless or low-quality data, thereby improving the overall accuracy and practicality of the data. Data credibility evaluation is a dynamic process that can be continuously updated and adjusted based on newly collected data and information to ensure continuous improvement of data quality. The sub-module effectively avoids the one-sidedness of single-dimensional evaluation, and can dynamically adjust data credibility as new data is added and old data is updated, ensuring the persistence and stability of data quality. The sub-module can adaptively adjust evaluation indicators and weights according to different data sets and task requirements to adapt to the diversity and complexity of Internet big data. By monitoring and updating data in real time, the sub-module can ensure the real-time and accuracy of data credibility, providing technical support for data extraction and decision-making. The sub-module improves the precision and efficiency of data screening, effectively excluding useless or low-quality data, and enhances the accuracy and reliability of the data set, providing strong support for subsequent data analysis and application.

[0062] More specifically: the sub-module not only considers the credibility of a single attribute, but also integrates the credibility of multiple attributes through weighted summation, making the evaluation result more comprehensive and accurate. The sub-module can consider the time factor of data and appropriately penalize outdated data, thereby improving the timeliness of the evaluation result. It can also filter out a data subset with higher credibility, effectively excluding the interference of useless information such as outdated information and phishing website information, and improving the accuracy and practicality of the data. During data extraction and processing, data can be sorted or screened according to the value of CVB(DA i ), thereby optimizing the data processing flow and improving processing efficiency.

[0063] Please refer to Figures 1 to 3 , the feature extraction sub-module processing process is as follows:

[0064]

[0065] wherein:

[0066] FGH(DB i ) refers to the optimized feature vector;

[0067] DB i refers to the data with high credibility filtered out from CVB(DB i ), FAs refers to the feature values of data DB j , FA i j refers to the jth feature value of data DB i , CVB(DB i ) refers to the credibility of data DB i , m refers to the total number of features of data DB i , FAs refers to the average value of all feature values of data DB j , FA i x log(l+CVB(DB i )) refers to weighting each feature value with the weight of log(l+CVB(DB i )) to emphasize the importance of data with high credibility in feature extraction, refers to the standard deviation of the feature vector, indicating the degree of dispersion of the feature values, refers to normalizing the credibility to smooth its impact on feature optimization;

[0068] The processing procedure of the feature extraction submodule is as follows: input the data DB i with high credibility to the feature extraction submodule, and output the optimized feature vector FGH(DB i ) based on the credibility CVB(DB i ) of data DB i and the jth feature value FA j of data DB i ;

[0069] The optimized data is DC i , that is, the data of DB i after optimization by the feature extraction submodule.

[0070] In this embodiment: the feature optimization process makes the features of certain data points more prominent or more in line with certain standards, thereby indirectly affecting subsequent screening. The calculation of this submodule can improve the discrimination and effectiveness of the feature vector, providing more accurate information for subsequent comprehensive evaluation. This submodule can efficiently extract key features from a large amount of raw data and further refine these features through optimization algorithms to form a more compact and representative feature set FGH(DB iThis greatly improves the efficiency and accuracy of data processing, making subsequent data analysis and decision-making processes more reliable. The feature extraction and optimization process can adaptively adjust parameters and methods to adapt to different types of datasets and task requirements, demonstrating strong flexibility and adaptability. This submodule adopts advanced feature extraction algorithms, which can quickly and accurately extract key features from a large amount of raw data. Through optimization methods such as dimensionality reduction, noise reduction, and feature selection, this submodule can flexibly adjust algorithms and parameters according to different feature extraction needs, realizing the scalability and flexibility of feature extraction. This submodule optimizes the data processing flow, reduces the complexity and cost of data processing, and improves the expressive power and discriminative power of data features, which helps to discover potential patterns and regularities in the data.

[0071] More specifically: This submodule introduces data credibility CVB (DB) i As weights, the feature values ​​are weighted to make the optimized feature vector more consistent with the actual situation of the data. This submodule can take into account the degree of dispersion between feature values ​​and further optimize and adjust the feature vector. The optimized feature vector can more accurately describe the feature information of the data and provide a richer data foundation for subsequent data analysis and mining.

[0072] Please see Figures 1 to 3 The evaluation and analysis submodule processing procedure is as follows:

[0073]

[0074] in:

[0075] SDF(DC i () refers to the comprehensive evaluation value of the data;

[0076] DC i Refers to the i-th data after feature optimization, CVB(DC) i (DC) refers to data i Credibility, SDQ j Refers to the standard deviation of the j-th feature, used to normalize the eigenvalues, FGH(DC) i (DC) refers to data i Optimized feature vectors, FFS j Refers to the j-th eigenvalue in the target eigenvector;

[0077] The evaluation and analysis submodule's processing procedure is as follows: Data DC i Input to the evaluation and analysis submodule, and correspondingly convert the data DC. i Data credibility CVB (DC) i ) and data DC iThe optimized feature vector FGH(DC i ) is input into the evaluation analysis submodule, and the comprehensive evaluation value SDF(DC i ) of the output data is output.

[0078] In this embodiment: this submodule combines the data reliability and the optimized feature vector to perform final screening to obtain a data subset that best meets the application requirements, so as to ensure the quality and accuracy of the final data subset and provide a reliable basis for subsequent analysis, mining or application. This submodule comprehensively considers data reliability, feature extraction results and other possible evaluation indexes, which helps to make more intelligent decisions during data extraction and ensures that the final obtained data set is both accurate and practical. The comprehensive evaluation and decision-making process can take into account multiple dimensions and factors to achieve comprehensive optimization and screening of data, providing a solid foundation for subsequent data analysis and application. This submodule comprehensively considers data reliability, feature extraction results and other evaluation indexes, and realizes comprehensive evaluation and optimization of data through complex mathematical models and algorithms. Based on the comprehensive evaluation result, this submodule can intelligently select the optimal data item to ensure the high quality and practicality of the final data set. The calculation of this submodule can cope with noise and abnormal data in Internet big data, maintain the stability and reliability of the evaluation result, and SDF(DC i ) can flexibly adjust the evaluation indexes and decision rules according to different evaluation requirements and decision targets to realize the customizability of evaluation and decision-making, realize comprehensive evaluation and optimization of data, ensure the high quality and practicality of the final data set, provide a scientific basis for decision-making in the data extraction process, and improve the efficiency and effectiveness of data extraction.

[0079] More specifically: this submodule not only considers the data reliability, but also combines the similarity between the optimized feature vector and the target feature vector for evaluation, so that the evaluation result is more comprehensive and accurate. By introducing the standard deviation SDQ j for normalization processing, this submodule can eliminate the dimensional differences between different feature values, so that the evaluation result is more fair and reasonable. By calculating SDF(DC i ), the data can be comprehensively evaluated and sorted or screened according to the evaluation value, thereby providing strong support for subsequent decision-making. In the case of limited resources, the data can be prioritized according to the value of SDF(DC i ), thereby optimizing resource allocation and improving resource utilization efficiency.

[0080] Furthermore, based on SDF(DC i ), the weight of the jth attribute in the data reliability evaluation submodule is CAA jThe cycle iteration is performed, so that the multiple groups of sub-modules in the method form a form of continuous cycle and optimization feedback, and the processing process is as follows:

[0081] Step one: CAA j，new = CAA j，old + a x SDF(DC i );

[0082] Step two: establish a cycle termination condition;

[0083] Condition one: | CAA j，new - CAA j，old | < CS;

[0084] Condition two: the number of iterations is 100 times;

[0085] Wherein:

[0086] CAA j，new refers to the weight of the jth attribute after iteration;

[0087] CAA j，old refers to the weight of the jth attribute before iteration, a refers to the learning rate, and CS refers to the convergence threshold 0.001;

[0088] In this embodiment: this iteration form can automatically adjust the weight parameter CAA j according to the actual performance of the data, so that the weight is more in line with the actual situation of the data, thereby improving the calculation accuracy of the data reliability, and through the iteration form, the optimal solution can be gradually approached, that is, a set of weight parameters CAA j is found, so that the calculation result of the data reliability is more accurate and stable. Since the algorithm can adaptively adjust the weight parameter, it has a certain robustness to data fluctuations and noise, and can better cope with complex data environments. By optimizing the weight parameter, the data reliability can be more accurately calculated, so as to filter out higher quality data and provide strong support for subsequent data analysis and processing. The algorithm can adaptively adjust the weight parameter according to different data characteristics and application scenarios, so it has stronger adaptability and flexibility. Through the iteration form, the optimal solution can be gradually approached, thereby reducing unnecessary calculation overhead and improving the efficiency of the algorithm. By improving the accuracy and efficiency of data extraction, this iteration form can promote the development of Internet big data analysis, data mining and other fields, and provide more reliable data support for related research and application. This iteration form provides a new idea and method for algorithm innovation and optimization, and helps to promote the technological innovation and progress in related fields. In practical applications, this iteration form can improve the effect of data extraction and analysis, and provide more accurate and reliable data support for enterprise decision-making and business development.

[0089] In the specific implementation process, the plurality of sub-modules in the method is used to constitute an Internet big data extraction system, the data set DA is input into the credibility sub-module, the i th data DA in the data set DA is selected according to the data set DA i , and the output data credibility CVB(DA i ) is output. The sub-module can remove low credibility data to reduce noise and interference, help to improve the efficiency and accuracy of subsequent processing, and high credibility data is more likely to contain valuable information. The sub-module can significantly improve the accuracy of data screening, can automatically identify and exclude a large amount of expired information, phishing website information and other useless or low-quality data, and improve the accuracy and practicability of the data as a whole. The sub-module effectively avoids the one-sidedness of single-dimensional evaluation, and can dynamically adjust the data credibility with the addition of new data and the update of old data, to ensure the continuity and stability of data quality.

[0090] The data DB i with high credibility is input into the feature extraction sub-module to output the optimized feature vector FGH(DB i ). In the feature optimization process, the features of some data points are more prominent or more in line with certain standards, thereby indirectly affecting subsequent screening. The calculation of the sub-module can improve the discrimination and effectiveness of the feature vector, and provide more accurate information for subsequent comprehensive evaluation. The sub-module can efficiently extract key features from a large amount of original data, and further refine these features through an optimization algorithm to form a more compact and representative feature set FGH(DB i ), thereby improving the efficiency and accuracy of data processing and making the subsequent data analysis and decision-making process more reliable. The feature extraction and optimization process can adaptively adjust parameters and methods to adapt to different types of data sets and task requirements, and exhibit strong flexibility and adaptability.

[0091] The data DC i is input into the evaluation analysis sub-module to output the comprehensive evaluation value SDF(DC i ) of the data. The sub-module combines data credibility and optimized feature vectors for final screening to obtain a data subset that best meets application requirements, to ensure the quality and accuracy of the final data subset and provide a reliable basis for subsequent analysis, mining or application. The sub-module comprehensively considers the credibility of the data and the results of feature extraction, which helps to make more intelligent decisions in the data extraction process and ensures that the final data set is both accurate and practical. The comprehensive evaluation and decision-making process can take into account multiple dimensions and factors to achieve comprehensive optimization and screening of data, and provide a solid foundation for subsequent data analysis and application.

[0092] Based on SDF(DC i ), the weight CAA jThe cyclic iteration is performed, so that the multiple sets of sub-modules in the method form a form of continuous circulation and optimization feedback, and the iteration form can automatically adjust the weight parameter CAA according to the actual performance of the data j , so that the weight is more in line with the actual situation of the data, thereby improving the data reliability calculation accuracy, and through the iterative form, the optimal solution can be gradually approached, that is, the more accurate weight parameter CAA is calculated j . The calculation result of the data reliability is more accurate and stable. Since the algorithm can adaptively adjust the weight parameter, it has a certain robustness to data fluctuations and noise, and can better cope with complex data environments. By optimizing the weight parameter, the data reliability can be more accurately calculated to filter out higher quality data to provide strong support for subsequent data analysis and processing. Through the iterative form, the optimal solution can be gradually approached to reduce unnecessary calculation overhead and improve the efficiency of the algorithm. Through improving the accuracy and efficiency of data extraction, this iterative form can promote the development of Internet big data analysis, data mining and other fields, and provide more reliable data support for related research and application.

[0093] Furthermore, the overall multiple sub-modules are calculated in cooperation with each other, and the overall circulation and iteration are also performed, so that the overall system has the effects of automatic optimization and updating, and the adaptability is better.

[0094] Embodiment two: please refer to Figure 1 , Figure 2 and Figure 3 , the data acquisition module uses the network crawler technology to crawl the target data on the website, and the crawler software runs on a high-performance server to ensure that a large number of data requests can be efficiently processed. Proxy IP pool and delay mechanism are used to avoid excessive pressure on the target website and being shielded by the anti-crawler mechanism. The target data collected by the acquisition module is specifically text, pictures, videos and tables, so as to ensure that the collected data covers different fields and topics, increases the diversity and comprehensiveness of the data, and the data in the processing module is also de-duplicated and normalized to ensure the uniqueness of each data.

[0095] In the embodiment: the data acquisition module and the processing module are used to crawl the target data from the network and upload and store the data, and process and clean the data for subsequent calculation and analysis, thereby playing a basic role for subsequent calculation and analysis and providing a large amount of data support for subsequent calculation and analysis.

[0096] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for extracting big data from the Internet, characterized in that: The extraction process includes the following steps: Step 1: Use the data acquisition module to scrape target data from the website and input the target data into the processing module; Step 2: The target data is cleaned by the processing module to remove tags, advertisements and invalid characters, and the text data and image data are converted into structured data to output dataset DA; Step 3: Input the dataset DA into the calculation module. The calculation module first calculates the confidence level of each data point in the dataset DA and outputs the data confidence level CVB(DA). i Then, based on the credibility threshold, select the data database with high credibility. i , for DB i Optimize and output the optimized feature vector FGH(DB) i Then output the optimized data DC. i Through optimized data DC i The overall evaluation value of the output data SDF (DC) i According to the comprehensive evaluation value SDF (DC) i Sort the data and select the data with the highest comprehensive evaluation value as the final extraction result; Step 4: Store the final extracted results using the storage module for subsequent analysis and application; The calculation module includes: a data credibility assessment submodule, a feature extraction submodule, and an assessment and analysis submodule; The calculation formula for the data credibility assessment submodule is as follows: ； in: CVB(DA i () refers to the credibility of the data; DA i The i-th data point in the dataset refers to the data DA. i Total number of attributes, CAA j Refers to the weight of the j-th attribute, CAB j Refers to data DA i The j-th attribute value, CAB j，max Referencing attribute CAB j Maximum value in the entire dataset, CAC j The base credibility of the j-th attribute is referred to as TN, the current time is referred to as TP, and the data DA is referred to as TP. i timestamp, Refers to data DA i Overall credibility The time decay function is used to consider the time freshness of data. The processing procedure of the credibility submodule is as follows: Input the dataset DA into the credibility submodule, and process the data according to the i-th data DA in the dataset DA. i , and its included attribute DA i The j-th attribute value CAB j And based on the basic credibility CAC of the j-th attribute j Output data credibility CVB(DA) i ); According to CVB(DA) i The value of ) is used to filter out data with high credibility in the DB database. i ; The calculation formula for the feature extraction submodule is as follows: ； in: FGH(DB i ) refers to the optimized feature vector; DB i Refers to CVB (DA) i Highly reliable data selected from ) and FA j Refers to the database (DB) i The j-th eigenvalue, CVB(DB) i ) refers to the database (DB) i Credibility, m refers to the data DB i Total number of features, FAs refer to data DB i The average of all eigenvalues, This refers to weighting each feature value, with the weight being log(1+CVB(DB)). i This emphasizes the importance of highly reliable data in feature extraction. The standard deviation of an eigenvector represents the degree of dispersion of its eigenvalues. This refers to normalizing the credibility to smooth its impact on feature optimization. The processing procedure of the feature extraction submodule is as follows: Data with high reliability is processed by DB. i Input is fed into the feature extraction submodule and based on the data DB. i Credibility of CVB (DB) i ) and data DB i The j-th eigenvalue FA j Output optimized feature vector FGH(DB) i ); The optimized data is DC i That is, DB i Data optimized by the feature extraction submodule.

2. The method for extracting big data from the Internet according to claim 1, characterized in that: The calculation formula for the evaluation and analysis submodule is as follows: ； in: SDF(DC i () refers to the comprehensive evaluation value of the data; DC i Refers to the i-th data after feature optimization, CVB(DC) i (DC) refers to data i Credibility, SDQ j Refers to the standard deviation of the j-th feature, used to normalize the eigenvalues, FGH(DC) i (DC) refers to data i Optimized feature vectors, FFS j Refers to the j-th eigenvalue in the target eigenvector; The processing procedure of the evaluation and analysis submodule is as follows: Data DC i Input to the evaluation and analysis submodule, and correspondingly convert the data DC. i Data credibility CVB (DC) i ) and data DC i Optimized feature vector FGH(DC) i The data is input into the evaluation and analysis submodule, and the output data is the comprehensive evaluation value SDF (DC). i ).

3. The method for extracting big data from the Internet according to claim 1, characterized in that: The data acquisition module uses web crawler technology to crawl target data from websites, and the crawler software runs on a high-performance server to ensure efficient processing of large numbers of data requests. It uses a proxy IP pool and a delay mechanism to avoid putting too much pressure on the target website and being blocked by anti-crawler mechanisms.

4. The method for extracting big data from the Internet according to claim 3, characterized in that: The data acquisition module captures specific data, including text, images, videos, and tables, to ensure that the collected data covers different fields and themes, increasing the diversity and comprehensiveness of the data.

5. The method for extracting big data from the Internet according to claim 1, characterized in that: The processing module also performs deduplication and normalization on the data to ensure the uniqueness of each data entry.

Images