A big data processing method, system and storage medium

Abstract

This invention provides a big data processing method, system, and storage medium, belonging to the field of big data processing and analysis technology. The method includes: acquiring raw data and classifying it through a data type identification module to generate a data type label sequence; performing preliminary cleaning on the raw data based on the data type label sequence to form a preliminarily cleaned dataset; generating a data quality score sequence based on the preliminarily cleaned dataset using a multi-dimensional data quality assessment model; identifying outliers and noise according to the data quality score sequence and repairing them to generate a repaired dataset after quality assessment; and performing multi-source data fusion and deduplication operations on the repaired dataset to form a deduplicated fused dataset. This invention improves the quality and accuracy of scientific and technological innovation big data, achieves efficient data processing and unified storage, and is suitable for processing complex multi-source data scenarios.

Description

Technical Field

[0001] This invention relates to the field of big data processing, and in particular to a big data processing method, system, and storage medium. Background Technology

[0002] With the rapid development of big data technology, especially in scientific research, technological innovation, and enterprise management, the scale and complexity of scientific and technological big data are growing exponentially. However, this data often comes from different data sources, including experimental data, sensor data, literature data, and historical data. These data not only differ in structure, format, and timeliness, but are also often affected by noise, missing values, outliers, and other issues, resulting in low data quality and seriously affecting the accuracy and efficiency of subsequent analysis and applications.

[0003] Traditional data processing methods typically rely on manual intervention or rule-based cleaning processes, which suffer from low processing efficiency, poor cleaning results, and high labor costs. Furthermore, since data originates from diverse sources, effectively integrating information from multiple data sources and removing redundant information to ensure data uniformity and consistency is a significant challenge currently facing technology. Therefore, improving the quality and accuracy of scientific and technological innovation big data through automated cleaning and multi-source data fusion technologies has become an urgent technical problem to be solved.

[0004] Existing solutions are mostly limited to cleaning data from a single data source, lacking a unified processing framework for multi-source data. Furthermore, they fail to adequately consider the correlations and heterogeneity between data points during the cleaning process, limiting their effectiveness when dealing with large-scale, high-dimensional science and technology innovation data. Therefore, how to efficiently clean data from multiple data sources based on automated algorithms, and achieve effective data fusion and deduplication while ensuring data quality, has become a key technical issue for improving the processing capabilities of science and technology innovation big data. Summary of the Invention

[0005] This invention provides a big data processing method, system, and storage medium to address how to improve the quality and accuracy of scientific and technological innovation big data through automated cleaning and multi-source data fusion technologies, ensuring that data from different data sources can be processed and stored efficiently and uniformly, and providing reliable basic data for subsequent data analysis, decision support, and intelligent applications.

[0006] To address the aforementioned technical problems, this invention provides a big data processing method, comprising:

[0007] The raw data is acquired and classified based on the data type identification module to obtain a data type label sequence;

[0008] Based on the data type label sequence, the original data is initially cleaned, including noise removal and missing value filling, to generate a preliminarily cleaned dataset.

[0009] Based on the pre-cleaned dataset, a multi-dimensional data quality assessment model is used to evaluate the integrity, accuracy, and consistency of the data, generating a data quality score sequence. According to the data quality score sequence, outliers and noise in the data are identified, and the outliers and noise are repaired to generate a repaired dataset after quality assessment.

[0010] Based on the repaired dataset, multi-source data fusion and deduplication are performed to remove duplicate data and integrate the cleaning results from different data sources to generate a deduplicated fused dataset.

[0011] The deduplicated fused dataset is stored in a designated storage medium and made available for subsequent data analysis and application through a data interface.

[0012] Furthermore, the step of conducting a data quality assessment based on the initially cleaned dataset includes:

[0013] The completeness of data items is assessed using a missing rate calculation formula, and the missing rate of each data item is calculated. ;

[0014] Based on standard score Z and Kullback-Leibler divergence Assess the accuracy of the data;

[0015] The consistency between data items is evaluated using a weighted cosine similarity and time alignment error model, generating a consistency score.

[0016] Furthermore, the step of generating the data quality score sequence includes:

[0017] Based on the comprehensive assessment results of completeness, accuracy, and consistency, a quality score is generated for each data point using a weighted calculation formula. ;

[0018] Through weighting coefficients , , Determine the contribution ratio of completeness, accuracy, and consistency to the data quality score, and generate a quality score sequence. .

[0019] Furthermore, the step of obtaining the raw data includes:

[0020] Raw data is obtained from multiple data sources, including sensor data, historical data, experimental data, and literature data.

[0021] Furthermore, the step of automatically classifying data based on the data type identification module includes:

[0022] By using preset classification rules or machine learning algorithms, the type of the original data is automatically identified and labeled, and a label sequence corresponding to the data type is generated.

[0023] Furthermore, the preliminary cleaning step includes:

[0024] The original data is processed using a denoising algorithm to remove noise data, which refers to data with abnormal fluctuations or deviations from the normal pattern.

[0025] Missing values ​​in data are filled using imputation algorithms, including interpolation or machine learning-based missing value prediction methods.

[0026] Furthermore, the data quality assessment steps include:

[0027] Based on statistical models or data distribution models, the pre-cleaned dataset is evaluated to identify outliers, noise, and inconsistent data, and a data quality report is generated.

[0028] Furthermore, the deduplicated fused dataset stored in the specified storage medium includes:

[0029] The cleaned and deduplicated data dataset contains multi-source data including sensor data, historical data, and experimental data, and stores metadata related to the data, including data source, timestamp, and cleaning status.

[0030] Furthermore, the efficient data output interface includes:

[0031] It provides a standardized API interface that supports multiple data output formats, including CSV, JSON, and XML.

[0032] Furthermore, a science and technology innovation big data processing system based on automated cleaning, applied to any of the above-mentioned big data processing methods, includes:

[0033] The data acquisition module (10) acquires data from multiple data sources in the field of science and technology innovation.

[0034] The data cleaning module (20) preprocesses the collected raw data.

[0035] The multi-source data fusion and deduplication module (30) merges cleaned data from different data sources and removes duplicate data items.

[0036] The data storage module (40) stores the deduplicated fused dataset in the specified storage medium.

[0037] The data output and interface module (50) provides an efficient data output interface.

[0038] The data analysis and decision support module (60) performs in-depth analysis based on the deduplicated fused dataset and generates corresponding reports or decision support information.

[0039] The system monitoring and adaptive module (70) continuously monitors the operating status of the entire system and optimizes system parameters through an adaptive algorithm.

[0040] The key innovations of this invention include:

[0041] (1) Automated data cleaning algorithm: An automated data cleaning algorithm based on machine learning and rules was developed, which can intelligently identify and repair noise, missing values ​​and outliers in big data of science and technology innovation.

[0042] (2) Multi-source heterogeneous data processing: The algorithm is applicable to structured, semi-structured and unstructured data, and supports cleaning of various data types such as text, image, audio and video.

[0043] (3) Efficient data quality improvement: Through parallel computing and optimization algorithms, the efficiency and accuracy of data cleaning are improved, and the data quality is significantly improved.

[0044] This invention utilizes automated data cleaning technology and multi-source data fusion algorithms, combined with optimization modules in a cloud computing platform, to intelligently process and repair big data related to science and technology innovation. This effectively solves problems such as poor data quality, low processing efficiency, and scattered data sources inherent in traditional data cleaning methods. Compared to traditional data cleaning methods, this invention can monitor and automatically repair missing values, outliers, and noise in the data in real time, ensuring data integrity, accuracy, and consistency, thereby providing reliable foundational data for subsequent data analysis and intelligent decision-making. Simultaneously, the optimized dataset can be continuously updated through the cloud platform, forming a closed-loop feedback mechanism that further improves data processing efficiency, reduces manual intervention, and lowers data cleaning costs. This method not only significantly improves data quality and processing accuracy but also achieves intelligent, automated, and efficient data processing, promoting the intelligent application and precise analysis of big data in science and technology innovation. Attached Figure Description

[0045] Figure 1 A flowchart illustrating a big data processing method provided in an embodiment of this application;

[0046] Figure 2 This is a structural block diagram of a big data processing system provided in an embodiment of this application. Detailed Implementation

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0048] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0049] Example 1: Refer to Figure 1 This is a flowchart illustrating a big data processing method provided in an embodiment of the present invention. The process may include at least steps S100-S500:

[0050] S100: Obtain the raw data and automatically classify it based on the data type recognition module to obtain a data type label sequence;

[0051] S200. Based on the data type label sequence, perform preliminary cleaning on the original data and generate a preliminarily cleaned dataset;

[0052] S300. Based on the dataset after preliminary cleaning, perform data quality assessment, identify and repair outliers and noise, and obtain the repaired dataset after quality assessment.

[0053] S400. Based on the repaired dataset, perform multi-source data fusion and deduplication to remove duplicate data and integrate the cleaning results from different data sources to obtain a deduplicated fused dataset.

[0054] S500: Store the deduplicated fused dataset in a specified storage medium and provide an efficient data output interface for subsequent data analysis.

[0055] Step S100 includes at least steps S110-S130:

[0056] S110: Obtain the raw data and perform preliminary processing to obtain the initial dataset.

[0057] First, the system retrieves raw data from multiple data sources. Assume the data source set is... ,in It can be a database, sensor, document, web crawler, image acquisition device, or audio / video input device, etc.

[0058] Data formatting: All raw data collected from the data source is transformed into a standard data structure through a unified formatting process. For example:

[0059] Text data is converted into a standard text format (such as UTF-8 encoding).

[0060] Image data is converted into a digitized image matrix.

[0061] Audio data is converted into time-domain signals or frequency-domain features.

[0062] Video data can be converted into image sequences or feature vectors of video frames.

[0063] Generate the initial dataset: Assume the initial dataset is ,in This represents each original data item. Initial dataset It contains all the raw data obtained from different data sources. Each data item It can be structured, semi-structured, or unstructured data. This dataset will serve as the basis for subsequent analysis and processing.

[0064] S120: Analyze the initial dataset based on the data type recognition algorithm to obtain a data type label sequence.

[0065] First, the system uses a pre-trained machine learning model (such as a deep neural network, support vector machine, etc.) for each data item. The algorithm performs classification and recognition. It determines the type of data item by analyzing its features (such as word frequency in text, pixel values ​​in images, and spectral characteristics in audio).

[0066] Furthermore, assuming each piece of original data Recognized as a certain data type Data type It can be one of the following:

[0067]

[0068]

[0069]

[0070]

[0071] Furthermore, regarding the preliminary dataset After classifying each data item, a data type label sequence is obtained. in For data items The corresponding type tag.

[0072] For each data item Through classification model Extracting feature vectors Then use the model To predict its data type label :

[0073]

[0074] in, This indicates that the trained classification model For eigenvectors Classify the data to obtain data items. Type tags .

[0075] The result of this step is a sequence of data type labels. This label sequence will serve as input for subsequent data cleaning and processing.

[0076] S130: Post-process the data type label sequence to generate the final classification result.

[0077] First, the system first... Perform a consistency check to ensure that the label of each data item is accurate. If inconsistent or ambiguous labels are found for some data items, the system will correct them using the following methods:

[0078] Furthermore, if the system finds that some data items are labeled with vague tags such as "image / video," it will further analyze the characteristics of the data items to confirm their specific type. For example, it can determine whether the data item is an image or a video by analyzing the pixel features of the data item or the frame rate of the video.

[0079] For ambiguous labels, such as "audio / video", the system refines the labels through further feature analysis (such as spectrum analysis, video frame feature extraction, etc.) to ensure that the label of each data item is clear.

[0080] If the system cannot determine the type of a data item during automatic classification, or if a classification error occurs, a manual review mechanism will be introduced, whereby human reviewers will review and correct any uncertain data items. This process will ensure the data type label sequence. High accuracy.

[0081] After post-processing, the system finally generates an accurate, cleaned sequence of data type labels. .in, Represents each data item The final data type label.

[0082] This tag sequence This will serve as input for subsequent data cleaning and processing steps, ensuring that the data is correctly categorized and cleaned.

[0083] During post-processing, if the labels are uncertain or incorrect, the system will recalculate the labels of the data items through a refinement or correction process. Assuming the corrected labels are... ,but:

[0084]

[0085] in, This indicates the correction process, which ensures the accuracy of the labels by further analyzing the characteristics of the data items.

[0086] S200 includes at least steps S210-S230:

[0087] S210: Denoise the data based on the data type label sequence.

[0088] Based on the final data type label sequence generated in S130 The system then begins denoising the data. Different denoising algorithms are applied based on the characteristics of each data type.

[0089] Text data Noise reduction:

[0090] The system utilizes Natural Language Processing (NLP) techniques to remove noise words such as invalid characters, punctuation marks, stop words, and spelling errors from text. It can identify the most representative words in the text using the TF-IDF algorithm and then remove irrelevant noise words.

[0091] For each piece of text data The text after denoising It can be represented as:

[0092]

[0093] in, This indicates that NLP technology is applied to remove noise from the text, and the denoised text is returned.

[0094] Image data Noise reduction:

[0095] For image data, the system can use common image denoising algorithms, such as median filtering and mean filtering, to remove noise from the image.

[0096] For each image data The image after denoising It can be represented as:

[0097]

[0098] in, This indicates that an image denoising algorithm is applied to process image data and remove noise from the image.

[0099] Audio data (T) i (Audio) Denoising:

[0100] The system removes background noise from audio data and retains valid speech or audio information through techniques such as spectrum analysis and filtering.

[0101] For each audio data segment The denoised audio It can be represented as:

[0102]

[0103] in, This indicates that an audio denoising algorithm is applied to process audio data and remove background noise from the audio.

[0104] Video data Noise reduction:

[0105] For video data, the system uses image denoising technology combined with temporal filtering algorithms to remove noise from the video.

[0106] For each video data segment The denoised video It can be represented as:

[0107]

[0108] in, This indicates that a video denoising algorithm is applied to remove noise from the video.

[0109] Through the above denoising steps, the system generates a dataset after noise removal. ,in This represents the data item after noise reduction.

[0110] S220: Impute missing values ​​in data based on data type label sequences.

[0111] After completing the denoising process, the system further processes the denoised dataset. Missing values ​​are filled in.

[0112] Text data Missing value imputation: For text data, the system uses context-based imputation methods, such as word embedding models (e.g., Word2Vec or GloVe), or text generation models (e.g., GPT-3 or BERT), to infer and fill in missing words or sentence parts.

[0113] Image data (T) i Image missing value imputation: For image data, the system uses image inpainting algorithms (such as image interpolation, generative adversarial networks (GANs)) to restore the missing parts of the image.

[0114] For each image data If it contains missing regions, the filled image It can be represented as:

[0115]

[0116] in, This indicates that an image restoration algorithm is used to restore the missing parts of the image.

[0117] Audio data Missing value imputation: For audio data, the system uses algorithms such as time-domain interpolation or spectral interpolation to recover the missing audio parts.

[0118] For each audio data segment If it contains missing parts, the filled audio It can be represented as:

[0119]

[0120] in, This indicates that missing parts of the audio are filled in using time-domain interpolation or spectral interpolation algorithms.

[0121] Video data (T) i (Video) Missing value filling: For video data, the system uses time frame interpolation or video restoration technology to fill in missing frames based on information from previous and subsequent frames.

[0122] For each video data segment If it contains missing frames, the video will be filled in. It can be represented as:

[0123]

[0124] in, This indicates that time frame interpolation or video restoration techniques are used to fill in missing frames.

[0125] After imputing these missing values, the system generates a preliminarily cleaned dataset. The dataset already contains data after denoising and filling in missing values.

[0126] S230: Generate the pre-cleaned dataset and prepare for subsequent cleaning steps.

[0127] After S220 completes data imputation, the system generates a preliminarily cleaned dataset. This dataset has undergone preliminary cleaning steps such as denoising and missing value imputation, and is ready for subsequent more complex data cleaning and data fusion operations.

[0128] After that, the system... Perform a simple quality check to confirm the integrity of the cleaned data. The check includes:

[0129] Validity check: Confirm that the data conforms to the expected format and range.

[0130] Accuracy check: Confirm that the data matches the actual situation and that there are no obvious errors.

[0131] Consistency check: Confirms whether the data is consistent across different data items.

[0132] For any anomalous data (such as outliers or inconsistent formats), the system will trigger further cleaning operations. This anomalous data will be marked and passed to subsequent processing stages.

[0133] Ultimately, the generated The dataset, after initial cleaning, will be transferred to subsequent advanced data processing stages. Specifically, the dataset... This will serve as input for subsequent steps such as data deduplication and data standardization, preparing for further refined data cleaning.

[0134] Step S300 includes at least steps S310-S330:

[0135] S310: Perform data quality assessment based on the pre-cleaned dataset.

[0136] First, the system processes the pre-cleaned dataset. A comprehensive data quality assessment was conducted, focusing on three dimensions: data completeness, accuracy, and consistency, to calculate a quality score sequence for each data point. .

[0137] ① Data integrity assessment. For datasets Assess the missing rate of each data point. This is then compared to the integrity threshold τ of the entire dataset. The formula is as follows:

[0138]

[0139] in, Represents data items The missing rate; Represents data items The value of the j-th field; m is the total number of fields in the data; 1( ) is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

[0140] If Pi > τ, then mark This is low-integrity data.

[0141] ② Data accuracy assessment. For numerical data, the accuracy of the data is assessed using a combination of Z-score and distribution bias. The specific formula is as follows:

[0142]

[0143] in, Standard score represents the deviation of a data point from the mean; and These are the mean and standard deviation of the dataset, respectively. This is the Kullback-Leibler (KL) divergence, used to measure the difference between the data distribution P(x) and the expected distribution Q(x).

[0144] If | |>3 or If the value is greater than δ, then mark it. This is abnormal data.

[0145] ③ Data consistency assessment. For multi-source data, calculate the consistency score between the data. A joint evaluation formula based on weighted cosine similarity and time alignment error is adopted:

[0146]

[0147] in, Represents data items and Consistency score; and This refers to the value of the data item in the k-th field. and For the timestamp of the data item; and These are the weighting coefficients. This is the time-aligned decay factor.

[0148] like If <ζ, then the data is determined. and There is a consistency issue.

[0149] ④ Calculate the quality score sequence. Based on the comprehensive assessment results of completeness, accuracy, and consistency, calculate the quality score qiq_i for each data point. The formula is as follows:

[0150]

[0151] in, , , The weight coefficients for each dimension satisfy... + + =1.

[0152] Based on the above evaluation, a quality score sequence for each data point is generated. This serves as the input for subsequent repair and fusion operations.

[0153] S320: Identify and repair outliers.

[0154] In S310, we have already obtained the quality score of the data item. Next, in S320, based on Identifying and correcting outliers. The key to outlier detection is using... and (Interquartile Spacing) Method.

[0155] use Detect outliers in numerical data. If If the value is not found in the specified range, the data item is considered an outlier. In addition, the system also employs... The method for detecting outliers is as follows:

[0156]

[0157] in, and These are the first and third quartiles of the dataset, respectively. The value is less than or greater than If it is, then it is considered an outlier.

[0158] For detected outliers, the system uses interpolation methods to repair them. For example, linear interpolation is used to repair missing numerical data. :

[0159]

[0160] in, These are the interpolation coefficients. and To and Adjacent valid data.

[0161] For time series data, the system uses trend analysis methods to repair outliers. Assuming the time series data is... In this case, outliers can be corrected through regression analysis.

[0162] The corrected data has been updated as follows:

[0163]

[0164] And generate the quality assessment results after repair:

[0165]

[0166] S330: Identify and repair noisy data.

[0167] After the S320 completes the outlier repair, the system then processes the noisy data, typically using signal processing techniques to identify and repair the noisy data.

[0168] The system utilizes frequency domain analysis techniques to detect noise. Taking audio data as an example, the system uses Fourier transform to perform spectral analysis on the audio signal:

[0169]

[0170] in, Time-domain signal For frequency, It is a frequency domain signal.

[0171] For image data, the system uses wavelet transform to detect noise. The wavelet transform formula is:

[0172]

[0173] in, The original signal, It is a wavelet function.

[0174] For detected noise, the system repairs it using denoising algorithms. For example, it uses a denoising convolutional neural network (CNN) to repair the image data:

[0175]

[0176] in, The original noise data, This is the data after noise reduction.

[0177] For audio data, the system uses a filter to smooth and repair the signal. The filter output is:

[0178]

[0179] in, These are the filter coefficients. This is the input signal.

[0180] Step S400 includes at least steps S410-S430:

[0181] S410: Multi-source data fusion.

[0182] In S410, we perform multi-source data fusion based on the repair dataset D3D_3D3. Assume that in the previous steps, we obtained cleaned datasets from multiple data sources, as shown below:

[0183]

[0184]

[0185] The datasets originate from different data sources and have already undergone preliminary data cleaning (denoising, missing value imputation, and outlier repair). Our goal is to merge them into a unified dataset, providing a foundation for deduplication and integration steps.

[0186] The first step in multi-source data fusion is aligning cleaned data items from different data sources. Data item alignment is typically based on certain common characteristics (e.g., data item identifiers, timestamps, or other unique identifiers). For each data item... We need to ensure that they correspond in the feature space. For example, for time series data, they need to be aligned according to time.

[0187] This process can be achieved by merging all data sources to obtain a merged dataset.

[0188]

[0189] in, This represents the merged dataset result after cleaning all data sources.

[0190] Aligned data source collection The data may contain duplicate items; therefore, we need to use feature matching algorithms to determine whether these data items represent the same entity. For example, we can use cosine similarity to measure the similarity between data items.

[0191]

[0192] in, and Each is a data item and In the Values ​​in dimensions This refers to the dimension of the data item. If the similarity exceeds a set threshold, the two data items are considered to come from the same entity.

[0193] For data items considered identical (i.e., highly similar), we need to fuse their features into a single, unified data item. Fusion methods can include simple averages, weighted averages, or other data synthesis methods. For example, we can use a weighted average to fuse the values ​​of two data items:

[0194]

[0195] in, and These are weighting coefficients, which are typically determined based on the quality score or credibility of the data items.

[0196] S420: Data deduplication.

[0197] In S420, we base our dataset on the merged dataset. Perform data deduplication. The purpose of data deduplication is to ensure that each data item in the final dataset is unique, avoiding interference from redundant data in subsequent processing.

[0198] To identify duplicate data items, we need to compare the similarity of each data item with other data items in the dataset. For example, using the aforementioned cosine similarity metric, if the similarity between two data items exceeds a certain set threshold... If they are duplicate data items, then they are considered duplicate data items.

[0199]

[0200] In this case, the data item and These are considered duplicates. At this point, the system will mark them and prepare for deduplication.

[0201] For identified duplicate data items, the system will select one item to keep and discard the others based on certain rules. Typically, we determine which data item is more reliable based on its quality score Q or other relevant characteristics (such as a timestamp). For example, we might select the data item with the higher quality score as the one to keep and discard the others.

[0202] The specific operation can be described by the following formula:

[0203]

[0204] in, It is the deduplicated dataset.

[0205] S430: Integrate cleaning results.

[0206] In S430, we base our dataset on the deduplicated dataset. Perform data integration operations to generate the final merged dataset. The goal of data integration is to ensure that data items cleaned from multiple data sources can be merged according to certain rules to obtain a comprehensive and accurate dataset.

[0207] For each data item after deduplication The system integrates its features with other relevant data items. The integration method is typically based on the quality score of the data items, source credibility, or other weighting mechanisms. For example, a weighted average can be used to merge data items from different sources.

[0208]

[0209] Among them, the function Representation of feature integration methods, This represents the data item that has been merged.

[0210] The integrated data items will be aggregated to generate a final fused dataset. This dataset will serve as the basis for the next stage of processing, such as advanced data analysis or modeling. (Merged dataset) It is a high-quality dataset that has been deduplicated and integrates data items from different sources.

[0211]

[0212] Step S500 includes at least steps S510-S530:

[0213] S510: Store the dataset to the specified storage medium.

[0214] In step S510, we first deduplicatize the merged dataset. The data is stored in a specified storage medium. This storage medium can be a traditional database system (such as SQL or NoSQL databases) or a distributed storage system (such as Hadoop HDFS or cloud storage). The choice of storage medium will affect storage efficiency and subsequent processing performance, depending on the data volume and future query needs.

[0215] The dataset needs to be Format it into a format suitable for storage. For example, for structured data, we can use... This format is suitable for efficient storage and retrieval of large-scale data. For unstructured data, The format can be used as a flexible storage option.

[0216] For data items We will serialize its characteristics to ensure that its format is suitable for the requirements of the storage medium:

[0217]

[0218] The formatted dataset is stored in the specified storage medium. Assuming the data is stored in a distributed storage system, the stored procedure can be completed using either batch processing or stream processing.

[0219]

[0220] in, This indicates the selected storage medium.

[0221] S520: Data storage optimization.

[0222] In S520, we optimize the data storage process to ensure efficient use of storage media and provide fast access paths. Data storage optimization is not only about saving storage space, but also about efficiently supporting subsequent data analysis tasks.

[0223] First, the stored dataset Create an index to quickly locate data items during subsequent analysis. The index can be built based on the primary key or other key feature fields, such as timestamps or categories. For each data item... We need to build an index:

[0224]

[0225] To improve storage efficiency and retrieval performance, the dataset Data compression can be performed. Compressing data reduces storage space and improves I / O efficiency during queries. Furthermore, dividing the dataset into multiple partitions (based on factors such as time and location) can also improve access performance.

[0226]

[0227]

[0228] S530: Provides a high-efficiency data output interface.

[0229] In S530, we specify the dataset stored in the specified storage medium. Provide efficient data output interfaces. These interfaces will support subsequent data analysis tasks, such as data visualization and machine learning model training. The data interfaces not only need to support common data query languages ​​(such as SQL), but also need to support API interfaces and batch data transfer capabilities.

[0230] To facilitate subsequent data access, we provide a standardized set of data query interfaces. For example, you can use either the SQL interface or the NoSQL interface to access data stored in the database. Let's assume we use the SQL interface to query the dataset. The query operation is as follows:

[0231]

[0232] In addition to traditional database query methods, we also provide a RESTful API interface for the dataset to support other systems in accessing the data via API. Assuming the data interface is a REST API, the API call can be performed as follows:

[0233]

[0234] To support large-scale data output, a batch data transfer interface is provided. For example, datasets can be transferred... by or Batch output of the format to other systems or analysis tools:

[0235]

[0236] The key innovations of this invention include:

[0237] (1) Automated data cleaning algorithm: An automated data cleaning algorithm based on machine learning and rules was developed, which can intelligently identify and repair noise, missing values ​​and outliers in big data of science and technology innovation.

[0238] (2) Multi-source heterogeneous data processing: The algorithm is applicable to structured, semi-structured and unstructured data, and supports cleaning of various data types such as text, image, audio and video.

[0239] (3) Efficient data quality improvement: Through parallel computing and optimization algorithms, the efficiency and accuracy of data cleaning are improved, and the data quality is significantly improved.

[0240] This invention utilizes automated data cleaning technology and multi-source data fusion algorithms, combined with optimization modules in a cloud computing platform, to intelligently process and repair big data related to science and technology innovation. This effectively solves problems such as poor data quality, low processing efficiency, and scattered data sources inherent in traditional data cleaning methods. Compared to traditional data cleaning methods, this invention can monitor and automatically repair missing values, outliers, and noise in the data in real time, ensuring data integrity, accuracy, and consistency, thereby providing reliable foundational data for subsequent data analysis and intelligent decision-making. Simultaneously, the optimized dataset can be continuously updated through the cloud platform, forming a closed-loop feedback mechanism that further improves data processing efficiency, reduces manual intervention, and lowers data cleaning costs. This method not only significantly improves data quality and processing accuracy but also achieves intelligent, automated, and efficient data processing, promoting the intelligent application and precise analysis of big data in science and technology innovation.

[0241] Example 2: Figure 2 A structural block diagram of a big data processing system according to an embodiment of the present invention is shown. Figure 2 As shown, the structure may include:

[0242] The data acquisition module (10) is responsible for acquiring data from multiple data sources in the field of science and technology innovation, including but not limited to patent data, scientific research papers, technical reports, and experimental data. This data is collected in real time through web crawlers, open APIs, or internal enterprise database interfaces, ensuring the breadth and diversity of data sources. Specifically, the data acquisition module collects data in different formats, such as text, images, and tables, through automated processes, providing rich input data for subsequent data processing.

[0243] The data cleaning module (20) performs preprocessing on the collected raw data, including noise reduction, missing value imputation, and outlier repair. This module uses advanced automated cleaning algorithms, such as machine learning-based classification models, to identify and repair noise and missing values ​​in the data. The cleaned data can be stored in structured, semi-structured, or unstructured formats to ensure that the data quality meets the needs of subsequent analysis.

[0244] Denoising: A data denoising algorithm is used to identify and remove invalid or duplicate data items.

[0245] Missing value imputation: Depending on the characteristics of the data, interpolation, regression models or multiple imputation methods are used to fill in missing data.

[0246] Outlier repair: Identify and repair outliers using rule-based or statistical analysis methods to ensure data consistency.

[0247] The multi-source data fusion and deduplication module (30) is responsible for merging cleaned data from different data sources and removing duplicate data items. This module uses feature matching algorithms, such as cosine similarity measurement, to determine the similarity between data items, identify duplicate entities, and generate unique fused data items through fusion strategies (such as weighted average).

[0248] Data alignment and fusion: Aligning data from multiple data sources to ensure that the same entities from different sources can be matched in the feature space.

[0249] Deduplication: Compare the similarity of data items, remove duplicate data items whose similarity exceeds a set threshold, and retain unique data items with higher quality.

[0250] The data storage module (40) is responsible for storing the deduplicated fused dataset in the specified storage medium. This module uses an efficient database system, supporting efficient access and querying of large-scale data. To adapt to future data analysis needs, the data storage module can provide a flexible storage structure, supporting incremental updates and backups of data.

[0251] Database support: Provides support for relational databases, NoSQL databases, and big data storage platforms (such as Hadoop, Spark, etc.).

[0252] Efficient Query: Utilizing indexing technology and optimized storage structure to ensure efficient data retrieval and analysis operations.

[0253] The data output and interface module (50) provides an efficient data output interface for subsequent data analysis and modeling. This module includes functions such as data export, API interface, and data visualization, and can provide clear and accurate data output for users or other systems. The output data can be used for further decision support, algorithm modeling, report generation, and other purposes.

[0254] Data Export: Supports exporting common data formats such as CSV, JSON, and XML, making it convenient for users and other systems to use.

[0255] API Interface: Provides a RESTful API interface to support integration with other systems or applications, ensuring real-time data sharing and retrieval.

[0256] The data analysis and decision support module (60) is responsible for performing in-depth analysis on the deduplicated fused dataset and generating corresponding reports or decision support information. This module utilizes machine learning and data mining techniques to extract valuable patterns and trends from the data, providing users with scientific and reasonable decision-making basis.

[0257] Data mining and modeling: Using techniques such as regression analysis, cluster analysis, and classification algorithms, we delve into the potential patterns in the data.

[0258] Report generation: Based on the analysis results, decision reports are automatically generated to help users make accurate decisions.

[0259] The system monitoring and adaptive module (70) continuously monitors the operating status of the entire system and optimizes system parameters through adaptive algorithms to ensure that the system operates efficiently and stably under various complex environments. This module can automatically adjust data cleaning and fusion strategies to ensure the robustness and efficiency of the data processing process.

[0260] Real-time monitoring: Monitor the system's operating status, detect and report any potential faults or bottlenecks.

[0261] Adaptive optimization: Based on real-time feedback data, it automatically adjusts data processing and storage parameters to optimize system performance.

[0262] The following are its main beneficial effects:

[0263] (1) Automated classification and cleaning: The data type identification module automatically classifies the data, enabling customized processing for different types of data. Compared with traditional manual intervention, this greatly reduces human error and improves the efficiency and accuracy of data cleaning.

[0264] (2) Noise removal and missing value filling: In the initial data cleaning stage, noise is automatically removed and missing values ​​are filled, ensuring the integrity of the data for subsequent analysis. Compared with traditional methods, this avoids the bias caused by manual judgment and improves the integrity and consistency of the data.

[0265] (3) Data quality assessment and repair: Through a multi-level data quality assessment and repair mechanism, outliers and noise can be accurately identified, ensuring that the final dataset meets high quality standards. The automated processing of this process not only improves the accuracy of data repair but also significantly shortens the data cleaning time.

[0266] (4) Multi-source data fusion and deduplication: In the process of multi-source data fusion and deduplication, this invention can efficiently integrate data from different sources, remove duplicate data, and avoid the impact of redundant data on the analysis results, providing a more accurate and efficient foundation for data analysis. The automation and intelligence of this process greatly improves the efficiency of data processing and avoids the duplication and conflict problems that exist in traditional manual fusion.

[0267] (5) High-efficiency storage and output interface: By storing the deduplicated dataset in a specified storage medium and providing a high-efficiency data output interface, this invention not only ensures the stability of data storage, but also greatly improves the response speed of subsequent data analysis and intelligent applications.

[0268] This application provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it implements the method provided in this application.

[0269] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0270] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0271] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0272] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0273] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0274] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A big data processing method, characterized in that, The method includes: Raw data is obtained from multiple data sources and categorized to obtain a sequence of data type labels. The raw data includes databases, sensors, documents, web crawlers, image acquisition devices, or audio / video input devices. The categorization includes: Text data is converted into standard text format, image data is converted into digitized image matrices, audio data is converted into time-domain signals or frequency-domain features, and video data is converted into feature vectors of image sequences or video frames to generate the initial dataset. The initial dataset is classified by analyzing the features of the data items using a pre-trained machine learning model; The fuzzy labels are further refined through feature analysis to generate the final data type label sequence. Based on the data type label sequence, the original data is initially cleaned, including noise removal and missing value filling, to generate a preliminarily cleaned dataset. Natural language processing is applied to denoise text data, mean filtering and average filtering are applied to image data, spectral analysis is applied to denoise audio data, and temporal filtering is applied to denoise video data. For text data, word embedding models are used to fill in missing values; for image data, image interpolation and generative adversarial networks are used to fill in missing regions; and for audio data, temporal interpolation and spectral interpolation are used to fill in missing parts. Based on the pre-cleaned dataset, a multi-dimensional data quality assessment model is used to evaluate the integrity, accuracy, and consistency of the data, and generate a data quality score sequence. Calculate the integrity score, accuracy score, and consistency score for each data point to generate a quality score sequence; The expression for the integrity score is: ； in, Represents data items The missing rate; Represents data items The value of the j-th field; m is the total number of fields in the data; It is an indicator function that takes the value 1 when the condition is true and 0 otherwise; if Pi > τ, then it is marked. This is low-integrity data; The expression for the accuracy score is: ； in, Standard score represents the deviation of a data point from the mean; and These are the mean and standard deviation of the dataset, respectively. The Kullback-Leibler (KL) divergence measures the difference between the data distribution P(x) and the expected distribution Q(x); if Then mark This is abnormal data; Based on the data quality scoring sequence, outliers and noise in the data are identified, and the outliers and noise are repaired to generate a repaired dataset after quality assessment. use The system detects outliers in numerical data and uses interpolation methods to repair them; for time series data, it uses regression analysis to repair outliers. Based on the repaired dataset, multi-source data fusion and deduplication are performed to remove duplicate data and integrate the cleaning results from different data sources to generate a deduplicated fused dataset. The multi-source data fusion and deduplication process uses cosine similarity to measure the similarity between data items. The deduplicated fused dataset is stored in a designated storage medium and made available for subsequent data analysis and application through a data interface.

2. The big data processing method according to claim 1, characterized in that, The steps for obtaining the raw data include: Raw data is obtained from multiple data sources, including databases, sensors, documents, web crawlers, image acquisition devices, or audio / video input devices.

3. The big data processing method according to claim 1, characterized in that, The data quality assessment steps include: Data quality assessment includes three dimensions: data integrity, accuracy, and consistency.

4. A big data processing system, applied to the big data processing method according to any one of claims 1-3, characterized in that, include: The data acquisition module (10) acquires data from multiple data sources in the field of science and technology innovation; The data cleaning module (20) preprocesses the collected raw data; The multi-source data fusion and deduplication module (30) merges cleaned data from different data sources and removes duplicate data items; The data storage module (40) stores the deduplicated fused dataset in the specified storage medium; The data output and interface module (50) provides an efficient data output interface; The data analysis and decision support module (60) performs in-depth analysis based on the deduplicated fused dataset and generates corresponding reports or decision support information; The system monitoring and adaptive module (70) continuously monitors the operating status of the entire system and optimizes system parameters through an adaptive algorithm.

Images