Data processing system for obtaining labels

Abstract

The invention discloses a data processing system for obtaining labels, the data processing system for obtaining labels has comprehensive data processing capability, the system can process multi-modal data from a plurality of data sources simultaneously, such as structured data, unstructured texts, images and the like, various advanced preprocessing technologies are adopted, and the data processing capability is improved. The method includes abnormal value processing, smoothing, wavelet denoising and the like, greatly improves data quality, combines a statistical method, signal processing, image processing, natural language processing and other feature extraction technologies, can more comprehensively capture essential features of data, has high-precision and dynamically-optimized tag generation, uses an integrated learning method, and improves the accuracy of data processing. According to the method, the advantages of a plurality of classifiers are integrated, the accuracy of label generation is remarkably improved, particularly, when complex and fuzzy samples are processed, active learning and semi-supervised learning technologies are introduced, continuous optimization of labels is achieved, the label quality is greatly improved, and the requirement for manual intervention is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing system for acquiring tags. Background Art

[0002] In the current era of big data, data labeling is crucial for fields such as machine learning, artificial intelligence, and data analysis. Accurate and efficient data labeling can not only improve the training effect of the model, but also provide a reliable foundation for data mining and decision support. At present, data label acquisition technology mainly includes the following aspects:

[0003] 1. Manual labeling: This is the most traditional method, where humans directly classify and label data. For example, in the field of image recognition, researchers manually add labels to a large number of pictures, such as "cat", "dog", etc. Although this method is highly accurate, it is inefficient, costly, and difficult to cope with massive amounts of data.

[0004] 2. Rule-based automatic labeling: This method automatically labels data using a predefined set of rules. For example, in text classification, the document category can be determined based on the frequency and location of keywords. This method is efficient but lacks flexibility and is difficult to handle complex and ambiguous situations.

[0005] 3. Machine learning assisted labeling: Use machine learning algorithms, such as support vector machines (SVM), decision trees, etc., to train models to automatically label data. This method combines the advantages of artificial intelligence and can handle more complex data, but its performance is heavily dependent on the quality and quantity of training data.

[0006] 4. Crowdsourcing annotation: Distribute annotation tasks to a large number of online workers through the Internet platform. This method can quickly process large amounts of data, but quality control is a challenge and may involve data privacy issues.

[0007] 5. Semi-supervised learning annotation: This method combines a small amount of labeled data with a large amount of unlabeled data to train the model. It attempts to reduce the reliance on a large amount of labeled data, but the accuracy of the model may not be as good as fully supervised learning.

[0008] However, these existing technologies still have the following deficiencies in practical applications: 1. Limited data processing capabilities: Most existing systems can only process data of a single type or source, and it is difficult to cope with complex data environments with multiple modalities and sources. In the preprocessing stage, many systems lack comprehensive cleaning, denoising and standardization capabilities, which affects the quality of subsequent processing. 2. Inadequate feature extraction and selection: Existing methods often only use a single feature extraction technology, which cannot fully capture the multidimensional characteristics of the data. At the same time, the lack of an effective feature selection mechanism leads to high model complexity and poor generalization ability. 3. Low label generation accuracy and efficiency: Traditional methods are not accurate enough when generating labels, especially when dealing with samples with blurred boundaries. At the same time, there is a lack of optimization and feedback mechanisms for generated labels, making it difficult to continuously improve label quality. 4. Poor system integration and scalability: Most existing systems are designed for specific tasks, lack modular and standardized interfaces, and are difficult to integrate with other systems or expand to new application scenarios. When faced with massive data, storage and retrieval efficiency is low, and visualization capabilities are insufficient, which is not conducive to data analysis and decision support.

[0009] Therefore, based on the above deficiencies in the prior art, the present invention provides a data processing system for acquiring labels, develops a comprehensive system capable of processing multi-source and multi-modal data, improves the comprehensiveness of data utilization and the adaptability of the system; introduces more advanced feature engineering technologies, including automatic feature extraction and selection, to improve the expressiveness and generalization performance of the model; adopts advanced machine learning technologies such as ensemble learning and transfer learning to improve the accuracy and efficiency of label generation, and introduces a dynamic optimization mechanism to achieve continuous improvement of label quality; designs a modular and scalable system architecture, provides a standardized API interface, optimizes data storage and retrieval mechanisms, and enhances data visualization capabilities to support a wider range of application scenarios and deeper data analysis. Summary of the invention

[0010] The purpose of the present invention is to provide a data processing system for acquiring labels, so as to solve the problems existing in the existing data label processing method proposed in the above background technology.

[0011] To achieve the above object, the present invention provides the following technical solution: a data processing system for obtaining tags, comprising:

[0012] A data acquisition module, used for collecting multimodal raw data from multiple data sources;

[0013] Data preprocessing module, used to clean, denoise, normalize and standardize the raw data;

[0014] Feature extraction module, used to extract high-dimensional feature vectors from preprocessed data;

[0015] Feature selection module, used to select the most representative feature subset from the high-dimensional feature vector;

[0016] The label generation module is used to generate multi-level and multi-granular labels based on the selected feature subset;

[0017] The label optimization module is used to verify, correct and optimize the generated labels;

[0018] A data storage module, used to store the generated tags and related metadata in a distributed manner;

[0019] Data retrieval module, used to quickly retrieve and obtain tags and related data;

[0020] Visualization module, used to intuitively present data processing process and results.

[0021] Preferably, the data acquisition module comprises the following steps:

[0022] s1: Configure multiple data source interfaces, including but not limited to relational databases, NoSQL databases, file systems, and streaming data sources;

[0023] s2: Implement corresponding data reading and parsing methods for each data source;

[0024] s3: Set data collection scheduling strategies, including batch collection and real-time streaming collection;

[0025] s4: Perform preliminary format conversion and coding unification on the collected data;

[0026] s5: Generate a data collection log to record information such as data source, collection time, and data volume.

[0027] Preferably, the data preprocessing module comprises the following steps:

[0028] s1: Check the data type and handle outliers on the original data;

[0029] s2: Use the moving average method to smooth the time series data;

[0030] s3: Use wavelet transform method to denoise the data, where the wavelet transform function is:

[0031]

[0032] Among them, a is the scale factor, b is the translation factor, and ψ(t) is the wavelet mother function;

[0033] s4: Use the minimum and maximum normalization method to normalize numerical features:

[0034]

[0035] s5: Use the Zscore method to standardize the data:

[0036]

[0037] Among them, μ is the mean and σ is the standard deviation.

[0038] Preferably, the feature extraction module adopts a variety of feature extraction methods, including but not limited to:

[0039] Statistical methods: calculate statistics such as mean, variance, skewness, kurtosis, etc.

[0040] Signal processing-based methods: frequency domain features are extracted using Fourier transform;

[0041] Image processing-based method: Use SIFT (Scale Invariant Feature Transform) algorithm to extract image features;

[0042] Method based on natural language processing: Use TFIDF (TermFrequencyInverseDocumentFrequency) to extract text features.

[0043] Preferably, the feature selection module comprises the following steps:

[0044] s1: Calculate the importance score of each feature, using methods including but not limited to:

[0045] Correlation-based methods

[0046] Mutual Information Based Methods

[0047] Feature Importance Based on Tree Model

[0048] s2: Sort features according to their importance scores;

[0049] s3: Use wrapper methods (such as recursive feature elimination) for feature subset selection;

[0050] s4: Apply L1 regularization (Lasso) to perform feature sparsification, and its optimization goal is:

[0051]

[0052] Among them, X is the feature matrix, y is the target variable, w is the weight vector, and α is the regularization parameter.

[0053] Preferably, the label generation module adopts an integrated learning method, comprising the following steps:

[0054] s1: Build multiple base classifiers, including but not limited to decision trees, support vector machines, and neural networks;

[0055] s2: Use the Bagging method to train multiple base classifiers, each classifier is trained on a randomly sampled sub-dataset;

[0056] s3: Use the Boosting method (such as AdaBoost) to iteratively train the classifier, where the weight α of the kth classifier k The calculation is as follows:

[0057]

[0058] Among them, ε k is the weighted error rate of the k-th classifier;

[0059] s4: Combine the outputs of multiple classifiers and use weighted voting to generate the final label:

[0060]

[0061] Among them, h k (x) is the prediction result of the kth base classifier.

[0062] Preferably, the label optimization module comprises the following steps:

[0063] s1: Calculate the label confidence of each sample using an entropy-based method:

[0064]

[0065] Among them, m is the number of categories, p i is the probability of the i-th category;

[0066] s2: Actively learn low-confidence samples and select the most informative samples for manual labeling;

[0067] s3: Use semi-supervised learning methods to generate pseudo labels for unlabeled samples using labeled samples;

[0068] s4: Apply label smoothing technology to reduce the risk of overfitting:

[0069] y smooth =(1-α)*y+α / K,

[0070] Among them, α is the smoothing parameter, K is the total number of categories;

[0071] s5: Retrain the model regularly and update the label generation rules.

[0072] Preferably, the data storage module adopts a distributed storage architecture, including the following steps:

[0073] s1: Use consistent hashing algorithm to shard data:

[0074] shard=hash(key)mod N,

[0075] Where N is the number of shards;

[0076] s2: Implement data replication mechanism to ensure data reliability;

[0077] s3: Use BloomFilter to build a data index and quickly determine whether the data exists:

[0078] P falsepositive =(1-e -kn / m ) k ,

[0079] Where k is the number of hash functions, n is the number of inserted elements, and m is the size of the bit array;

[0080] s4: realize data compression storage, using differential encoding and run-length encoding;

[0081] s5: Regularly merge and compress data to optimize storage space utilization.

[0082] Preferably, the data retrieval module comprises the following steps:

[0083] s1: Build an inverted index to support multi-field combination query;

[0084] s2: Implement similarity search based on LSH:

[0085] P[h(x)=h(y)]=sim(x,y),

[0086] Among them, h is the hash function, sim(x, y) is the similarity function;

[0087] s3: Uses a cache mechanism to speed up access to hot data and adopts an LRU (Least Recently Used) strategy for cache updates;

[0088] s4: implements distributed query optimization, including query decomposition, parallel execution, and result merging;

[0089] s5: Provides a RESTful API interface that supports flexible query condition combinations and result filtering.

[0090] Preferably, the visualization module includes the following functions:

[0091] Data distribution visualization: Use histograms, box plots, etc. to display feature distribution;

[0092] Feature correlation visualization: Use heatmap to display the correlation coefficient matrix between features;

[0093] Dimensionality reduction visualization: Use the tSNE (tdistributed Stochastic Neighbor Embedding) algorithm for two-dimensional visualization of high-dimensional data:

[0094]

[0095] Decision Boundary Visualization: Plot the decision boundary of a classifier in a two-dimensional feature space;

[0096] Model performance visualization: Use ROC curves, PR curves, etc. to display model performance;

[0097] Label evolution visualization: Use Sankey diagram to show the label change trend over time.

[0098] Compared with the prior art, the present invention has the following beneficial effects:

[0099] 1. The data processing system for obtaining labels has comprehensive data processing capabilities. The system can simultaneously process multimodal data from multiple data sources, such as structured data, unstructured text, images, etc. It uses a variety of advanced pre-processing technologies, including outlier processing, smoothing, wavelet denoising, etc., which greatly improves data quality. It combines statistical methods, signal processing, image processing, natural language processing and other feature extraction technologies to more comprehensively capture the essential characteristics of data;

[0100] 2. The data processing system for obtaining labels has high-precision and dynamically optimized label generation. It uses an integrated learning method and combines the advantages of multiple classifiers to significantly improve the accuracy of label generation. Especially when dealing with complex and fuzzy samples, it introduces active learning and semi-supervised learning techniques to achieve continuous optimization of labels, greatly improve label quality, and reduce the need for manual intervention. It uses a variety of feature importance evaluation methods and feature selection algorithms to effectively reduce data dimensions and improve the generalization ability and computing efficiency of the model. The combination of these technologies not only improves the accuracy of labels, but also realizes dynamic optimization of label quality, greatly enhancing the practicality and reliability of the system;

[0101] 3. The data processing system for obtaining labels can realize efficient data management and retrieval. It adopts distributed storage architecture and advanced indexing technology, significantly improving the storage efficiency and retrieval speed of massive data. It provides RESTful API, supports flexible query and filtering operations, greatly improves the system's integrability and ease of use, and provides rich visualization functions such as data distribution, feature correlation, and decision boundaries, greatly enhancing data insight capabilities;

[0102] 4. The data processing system for acquiring labels has wide applicability and scalability. The modular design and distributed architecture of the system make the system easy to expand and upgrade, and can adapt to future changes in demand and technological development. It can be applied to multiple fields, such as financial risk control, medical diagnosis, intelligent manufacturing, etc., which greatly expands the application scope and market value of the system. Through automation and intelligent processing, the time and labor costs of data processing and label acquisition are significantly reduced, so that the system can not only meet current needs, but also have the ability to cope with future challenges, greatly improving the long-term value of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 This is an overall architecture diagram of the data processing system for acquiring tags of the present invention;

[0104] Figure 2 It is a data preprocessing flow chart of the present invention;

[0105] Figure 3 A flow chart for label generation and optimization of the present invention. DETAILED DESCRIPTION

[0106] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0107] In the description of the invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0108] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0109] See also Figure 1-3 The present invention provides a technical solution: a data processing system for obtaining a tag, comprising:

[0110] A data acquisition module, used for collecting multimodal raw data from multiple data sources;

[0111] Data preprocessing module, used to clean, denoise, normalize and standardize the raw data;

[0112] Feature extraction module, used to extract high-dimensional feature vectors from preprocessed data;

[0113] Feature selection module, used to select the most representative feature subset from the high-dimensional feature vector;

[0114] The label generation module is used to generate multi-level and multi-granular labels based on the selected feature subset;

[0115] The label optimization module is used to verify, correct and optimize the generated labels;

[0116] A data storage module, used to store the generated tags and related metadata in a distributed manner;

[0117] Data retrieval module, used to quickly retrieve and obtain tags and related data;

[0118] Visualization module, used to intuitively present data processing process and results.

[0119] Specifically, from data collection to final visualization, each module represents a key step in data processing. The system covers the entire process from data collection to result presentation, providing a one-stop solution. Each module can be optimized and upgraded independently, which improves the maintainability and scalability of the system. Users can selectively use certain modules as needed to adapt to different application scenarios. Through automated and intelligent processing, the efficiency of data processing and label acquisition is greatly improved.

[0120] The data acquisition module includes the following steps:

[0121] s1: Configure multiple data source interfaces, including but not limited to relational databases, NoSQL databases, file systems, and streaming data sources;

[0122] s2: Implement corresponding data reading and parsing methods for each data source;

[0123] s3: Set data collection scheduling strategies, including batch collection and real-time streaming collection;

[0124] s4: Perform preliminary format conversion and coding unification on the collected data;

[0125] s5: Generate a data collection log to record information such as data source, collection time, and data volume.

[0126] The data preprocessing module includes the following steps:

[0127] s1: Check the data type and handle outliers on the original data;

[0128] s2: Use the moving average method to smooth the time series data;

[0129] s3: Use wavelet transform method to denoise the data, where the wavelet transform function is:

[0130]

[0131] Among them, a is the scale factor, b is the translation factor, and ψ(t) is the wavelet mother function;

[0132] s4: Use the minimum and maximum normalization method to normalize numerical features:

[0133]

[0134] s5: Use the Zscore method to standardize the data:

[0135]

[0136] Among them, μ is the mean and σ is the standard deviation.

[0137] Specifically, it should be noted that denoising is to use methods such as wavelet transform to eliminate noise in data and improve data quality. Denoising can eliminate interference information and improve the signal-to-noise ratio of data; normalization is to scale data to a specific range (such as [0,1]) to eliminate the impact of dimensions. Through normalization and standardization, features of different scales can be directly compared; and standardization is to convert data into a distribution with a mean of 0 and a variance of 1, which is conducive to the application of certain machine learning algorithms. Standardized data can accelerate the convergence of optimization algorithms such as gradient descent. Therefore, the preprocessed data is more suitable for machine learning algorithms, which can improve the training effect and prediction accuracy of the model.

[0138] The feature extraction module uses a variety of feature extraction methods, including but not limited to:

[0139] Statistical methods: calculate statistics such as mean, variance, skewness, kurtosis, etc.

[0140] Signal processing-based methods: frequency domain features are extracted using Fourier transform;

[0141] Image processing-based method: Use SIFT (Scale Invariant Feature Transform) algorithm to extract image features;

[0142] Method based on natural language processing: Use TFIDF (TermFrequencyInverseDocumentFrequency) to extract text features.

[0143] Specifically, multimodal data processing is used, which can process various types of data such as structured data, images, text, etc. The combination of multiple methods can capture multiple dimensions of data and provide a more comprehensive feature representation. Different types of feature extraction methods are suitable for different data types and tasks, which enhances the versatility of the system. In addition, the rich features help improve the performance of subsequent machine learning models.

[0144] Specifically, the data processing system for acquiring labels has comprehensive data processing capabilities. The system can simultaneously process multimodal data from multiple data sources, such as structured data, unstructured text, images, etc. It adopts a variety of advanced preprocessing technologies, including outlier processing, smoothing processing, wavelet denoising, etc., which greatly improves the data quality. It combines statistical methods, signal processing, image processing, natural language processing and other feature extraction technologies to more comprehensively capture the essential characteristics of the data.

[0145] The feature selection module includes the following steps:

[0146] s1: Calculate the importance score of each feature, using methods including but not limited to:

[0147] Correlation-based methods

[0148] Mutual Information Based Methods

[0149] Feature Importance Based on Tree Model

[0150] s2: Sort features according to their importance scores;

[0151] s3: Use wrapper methods (such as recursive feature elimination) for feature subset selection;

[0152] s4: Apply L1 regularization (Lasso) to perform feature sparsification, and its optimization goal is:

[0153]

[0154] Among them, X is the feature matrix, y is the target variable, w is the weight vector, and α is the regularization parameter.

[0155] Specifically, feature importance scores can use methods such as correlation coefficient, mutual information, and feature importance based on tree models, which can reduce the number of features, reduce model complexity, and prevent overfitting; wrapper methods can use recursive feature elimination (RFE). Reducing the number of features can speed up model training and prediction; L1 regularization (Lasso) achieves feature selection by adding an L1 norm penalty term. Selecting the most relevant features helps understand the data and model, and removing irrelevant features can improve model performance.

[0156] The label generation module adopts an integrated learning method, which includes the following steps:

[0157] s1: Build multiple base classifiers, including but not limited to decision trees, support vector machines, and neural networks;

[0158] s2: Use the Bagging method to train multiple base classifiers, each classifier is trained on a randomly sampled sub-dataset;

[0159] s3: Use the Boosting method (such as AdaBoost) to iteratively train the classifier, where the weight α of the kth classifier k The calculation is as follows:

[0160]

[0161] Among them, ε k is the weighted error rate of the k-th classifier;

[0162] s4: Combine the outputs of multiple classifiers and use weighted voting to generate the final label:

[0163]

[0164] Among them, h k (x) is the prediction result of the kth base classifier.

[0165] Specifically, ensemble learning is a method of combining multiple learners to improve performance. Bagging (Bootstrap Aggregating) is to train multiple classifiers on different data subsets and then vote. Boosting is to iteratively train classifiers, focusing on previously misclassified samples each time, such as the AdaBoost algorithm. Weighted voting is to give different weights to each classifier based on its performance, and then comprehensively obtain the final result. In this way, integrating multiple classifiers can reduce the error of a single classifier and improve accuracy. Different base classifiers can capture different aspects of the data and improve the generalization performance of the model. In addition, the ensemble method is more robust to noise and outliers, and can better handle nonlinear and high-dimensional data.

[0166] Specifically, the data processing system for obtaining labels has high-precision and dynamically optimized label generation. It uses an integrated learning method and combines the advantages of multiple classifiers to significantly improve the accuracy of label generation. In particular, when processing complex and fuzzy samples, it introduces active learning and semi-supervised learning techniques to achieve continuous optimization of labels, greatly improve label quality, and reduce the need for manual intervention. It uses a variety of feature importance evaluation methods and feature selection algorithms to effectively reduce data dimensions and improve the generalization ability and computing efficiency of the model. The combination of these technologies not only improves the accuracy of labels, but also achieves dynamic optimization of label quality, greatly enhancing the practicality and reliability of the system.

[0167] The label optimization module includes the following steps:

[0168] s1: Calculate the label confidence of each sample using an entropy-based method:

[0169]

[0170] Among them, m is the number of categories, p i is the probability of the i-th category;

[0171] s2: Actively learn low-confidence samples and select the most informative samples for manual labeling;

[0172] s3: Use semi-supervised learning methods to generate pseudo labels for unlabeled samples using labeled samples;

[0173] s4: Apply label smoothing technology to reduce the risk of overfitting:

[0174] y smooth =(1-α)*y+α / K,

[0175] Among them, α is the smoothing parameter, K is the total number of categories;

[0176] s5: Retrain the model regularly and update the label generation rules.

[0177] Specifically, label confidence is usually calculated using an entropy-based method, which reflects the model's confidence in the prediction results. Active learning is a machine learning method that selects the most informative unlabeled samples for manual labeling to improve model performance. Active learning can minimize the workload of manual labeling. Semi-supervised learning is a method of training a model using a small amount of labeled data and a large amount of unlabeled data. Semi-supervised learning makes full use of a large amount of unlabeled data to improve model performance. Label smoothing is to reduce model overconfidence and improve generalization ability by adding a small amount of noise to the prediction. Label smoothing technology helps to reduce overfitting and improve the performance of the model on new data, which can improve the quality of labels. The accuracy of labels is continuously improved through multiple optimization mechanisms, and regular updates enable the model to adapt to changes in data distribution.

[0178] The data storage module adopts a distributed storage architecture, including the following steps:

[0179] s1: Use consistent hashing algorithm to shard data:

[0180] shard=hash(key)mod N,

[0181] Where N is the number of shards;

[0182] s2: Implement data replication mechanism to ensure data reliability;

[0183] s3: Use BloomFilter to build a data index and quickly determine whether the data exists:

[0184] P falsepositive =(1-e -kn / m ) k ,

[0185] Where k is the number of hash functions, n is the number of inserted elements, and m is the size of the bit array;

[0186] s4: realize data compression storage, using differential encoding and run-length encoding;

[0187] s5: Regularly merge and compress data to optimize storage space utilization.

[0188] Specifically, consistent hashing, as a distributed hashing method, can minimize data migration when nodes are added or reduced. Distributed architecture and consistent hashing make the system easy to expand; data replication is the storage of copies of the same data on multiple nodes, which improves data availability and reading performance. The data replication mechanism improves the fault tolerance of the system; BloomFilter is a probabilistic data structure with high space efficiency, which is used to determine whether an element is in a set. BloomFilter can quickly determine whether data exists and improve query speed; data compression methods such as differential encoding and run-length encoding reduce data storage space, and data compression and regular optimization significantly reduce storage space requirements; data merging and compression can regularly organize and optimize data to improve storage efficiency, and sharding and indexing mechanisms improve data read and write performance.

[0189] The data retrieval module includes the following steps:

[0190] s1: Build an inverted index to support multi-field combination query;

[0191] s2: Implement similarity search based on LSH:

[0192] P[h(x)=h(y)]=sim(x,y),

[0193] Among them, h is the hash function, sim(x, y) is the similarity function;

[0194] s3: Uses a cache mechanism to speed up access to hot data and adopts an LRU (Least Recently Used) strategy for cache updates;

[0195] s4: implements distributed query optimization, including query decomposition, parallel execution, and result merging;

[0196] s5: Provides a RESTful API interface that supports flexible query condition combinations and result filtering.

[0197] Specifically, inverted index, as an indexing method, stores a list of documents where each word appears, and is suitable for full-text search. Inverted index and LSH greatly improve the speed and flexibility of data retrieval; LSH (Locality-Sensitive Hashing) is an approximate nearest neighbor search algorithm used for similarity search of high-dimensional data, supports content-based similarity search, and expands the application scope of the system; caching mechanisms such as LRU (Least Recently Used) strategy store frequently used data in fast storage, and the caching mechanism significantly improves the access speed of hot data; distributed query optimization includes query decomposition, parallel execution, and result merging. Distributed query optimization improves the query efficiency of large-scale data sets; RESTful API is an HTTP-based API design style that provides a standardized interface. RESTful API makes the system easy to integrate with other applications.

[0198] The visualization module includes the following functions:

[0199] Data distribution visualization: Use histograms, box plots, etc. to display feature distribution;

[0200] Feature correlation visualization: Use heatmap to display the correlation coefficient matrix between features;

[0201] Dimensionality reduction visualization: Use the tSNE (tdistributed Stochastic Neighbor Embedding) algorithm for two-dimensional visualization of high-dimensional data:

[0202]

[0203] Decision Boundary Visualization: Plot the decision boundary of a classifier in a two-dimensional feature space;

[0204] Model performance visualization: Use ROC curves, PR curves, etc. to display model performance;

[0205] Label evolution visualization: Use Sankey diagram to show the label change trend over time.

[0206] Specifically, by intuitively displaying data features, it helps users understand data distribution and feature relationships. Decision boundary and performance visualization helps understand and optimize models. Dimensionality reduction visualization makes patterns and clusters of high-dimensional data visible. Label evolution visualization helps understand how data changes over time. Comprehensive visualization capabilities provide strong support for data analysis and decision-making.

[0207] Specifically, the data processing system for obtaining labels can realize efficient data management and retrieval. It adopts distributed storage architecture and advanced indexing technology, which significantly improves the storage efficiency and retrieval speed of massive data. It provides RESTful API and supports flexible query and filtering operations, which greatly improves the system's integrability and ease of use. It provides rich visualization functions such as data distribution, feature correlation, decision boundaries, etc., which greatly enhances data insight capabilities.

[0208] Specifically, the data processing system for acquiring labels has wide applicability and scalability. The modular design and distributed architecture of the system make the system easy to expand and upgrade, and can adapt to future changes in demand and technological development. It can be applied to multiple fields, such as financial risk control, medical diagnosis, intelligent manufacturing, etc., which greatly expands the application scope and market value of the system. Through automation and intelligent processing, the time and labor costs of data processing and label acquisition are significantly reduced, so that the system can not only meet current needs, but also have the ability to cope with future challenges, greatly improving the long-term value of the system.

[0209] Example 1: Financial Fraud Detection System

[0210] This embodiment describes an application of using a data processing system that acquires tags to perform financial fraud detection.

[0211] 1. Data Collection

[0212] The system collected 10,000 transaction records from a bank's transaction database. Each record contained the following features: transaction amount, transaction time, account age, recent transaction frequency, transaction location, etc.

[0213] 2. Data Preprocessing

[0214] a) Denoising: Use wavelet transform method to denoise the transaction amount.

[0215] Wavelet transform function:

[0216]

[0217] Use Haar wavelet as the mother wavelet function to transform a transaction amount of 100.5:

[0218] Assume a=2, b=0, then:

[0219]

[0220] This means that at this scale, the signal has no significant high-frequency noise.

[0221] b) Normalization: Perform minimum-maximum normalization on the transaction amount.

[0222]

[0223] Assuming the minimum transaction amount is 10 yuan and the maximum is 10,000 yuan, 100.5 yuan is normalized to:

[0224]

[0225] c) Standardization: Z-score standardization is performed on the account age.

[0226]

[0227] Assuming that the mean account age μ = 5 years, the standard deviation σ = 2 years, and the age of an account is 3 years, the standardized value is:

[0228]

[0229] 3. Feature extraction

[0230] The TF-IDF method is used to extract the characteristics of the transaction location. Assuming that a certain transaction location in a certain city appears 100 times in all transactions and the total number of transactions is 10,000, its TF-IDF value is:

[0231]

[0232] 4. Feature Selection

[0233] Use L1 regularization (Lasso) for feature selection. The optimization goal is:

[0234]

[0235] Suppose we have 5 features, and the weight vector w obtained after Lasso regression is [0.1, 0, 0.3, 0, 0.2], which means that the 2nd and 4th features are selected.

[0236] 5. Label generation

[0237] The labels are generated using the AdaBoost ensemble method. The weight of the kth classifier is calculated as follows:

[0238]

[0239] Assuming the error rate of the first classifier ε1 = 0.3, its weight is:

[0240]

[0241] The final classification result is determined by weighted voting:

[0242]

[0243] 6. Tag optimization

[0244] Calculate label confidence using the concept of entropy:

[0245]

[0246] Assuming that the probability of a sample being classified as fraudulent is p1 = 0.8 and the probability of non-fraud is p2 = 0.2, then its confidence is:

[0247]

[0248] 7. Data Storage

[0249] Use consistent hashing for data sharding:

[0250] shard=hash(key)mod N,

[0251] Assume there are 4 storage nodes, and the hash value of a transaction record is 12345, then its storage location is:

[0252] shard = 12345 mod 4 = 1,

[0253] The record will be stored on the second node (counting starts from 0).

[0254] 8. Data Retrieval

[0255] Use BloomFilter for fast retrieval. Assuming n = 1000 elements are inserted and the false positive rate is 0.01, the required bit array size m and the number of hash functions k are:

[0256]

[0257] 9. Visualization

[0258] Use the t-SNE algorithm for dimensionality reduction visualization. The core of t-SNE is to calculate the conditional probability between point pairs in high-dimensional space:

[0259]

[0260] Suppose in a high-dimensional space, two points x i and x j The Euclidean distance is 5, σ i =2, then the conditional probability between them is:

[0261]

[0262] In practice, this value is normalized to the distances of all other points.

[0263] Through this system, the bank successfully identified about 2% of fraudulent transactions, greatly reducing financial risks.

[0264] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing system for obtaining tags, characterized in that: include: A data acquisition module, used for collecting multimodal raw data from multiple data sources; Data preprocessing module, used to clean, denoise, normalize and standardize the raw data; Feature extraction module, used to extract high-dimensional feature vectors from preprocessed data; Feature selection module, used to select the most representative feature subset from the high-dimensional feature vector; The label generation module is used to generate multi-level and multi-granular labels based on the selected feature subset; The label optimization module is used to verify, correct and optimize the generated labels; A data storage module, used to store the generated tags and related metadata in a distributed manner; Data retrieval module, used to quickly retrieve and obtain tags and related data; Visualization module, used to intuitively present data processing process and results.

2. A data processing system for acquiring tags according to claim 1, characterized in that: The data acquisition module comprises the following steps: s1: Configure multiple data source interfaces, including but not limited to relational databases, NoSQL databases, file systems, and streaming data sources; s2: Implement corresponding data reading and parsing methods for each data source; s3: Set data collection scheduling strategies, including batch collection and real-time streaming collection; s4: Perform preliminary format conversion and coding unification on the collected data; s5: Generate a data collection log to record information such as data source, collection time, and data volume.

3. A data processing system for acquiring tags according to claim 1, characterized in that: The data preprocessing module comprises the following steps: s1: Check the data type and handle outliers on the original data; s2: Use the moving average method to smooth the time series data; s3: Use wavelet transform method to denoise the data, where the wavelet transform function is: Among them, a is the scale factor, b is the translation factor, and ψ(t) is the wavelet mother function; s4: Use the minimum and maximum normalization method to normalize numerical features: s5: Use the Zscore method to standardize the data: Among them, μ is the mean and σ is the standard deviation.

4. A data processing system for acquiring tags according to claim 1, characterized in that: The feature extraction module uses a variety of feature extraction methods, including but not limited to: Statistical methods: calculate statistics such as mean, variance, skewness, kurtosis, etc. Signal processing-based methods: frequency domain features are extracted using Fourier transform; Image processing-based method: Use SIFT (Scale Invariant Feature Transform) algorithm to extract image features; Method based on natural language processing: Use TFIDF (TermFrequencyInverseDocumentFrequency) to extract text features.

5. A data processing system for acquiring tags according to claim 1, characterized in that: The feature selection module comprises the following steps: s1: Calculate the importance score of each feature, using methods including but not limited to: Correlation-based methods Mutual Information Based Methods Feature Importance Based on Tree Model s2: Sort features according to their importance scores; s3: Use wrapper methods (such as recursive feature elimination) for feature subset selection; s4: Apply L1 regularization (Lasso) to perform feature sparsification, and its optimization goal is: Among them, X is the feature matrix, y is the target variable, w is the weight vector, and α is the regularization parameter.

6. A data processing system for acquiring tags according to claim 1, characterized in that: The label generation module adopts an integrated learning method, including the following steps: s1: Build multiple base classifiers, including but not limited to decision trees, support vector machines, and neural networks; s2: Use the Bagging method to train multiple base classifiers, each classifier is trained on a randomly sampled sub-dataset; s3: Use the Boosting method (such as AdaBoost) to iteratively train the classifier, where the weight α of the kth classifier k The calculation is as follows: Among them, ε k is the weighted error rate of the k-th classifier; s4: Combine the outputs of multiple classifiers and use weighted voting to generate the final label: Among them, h k (x) is the prediction result of the kth base classifier.

7. A data processing system for acquiring tags according to claim 1, characterized in that: The label optimization module includes the following steps: s1: Calculate the label confidence of each sample using an entropy-based method: Among them, m is the number of categories, pi is the probability of the i-th category; s2: Actively learn low-confidence samples and select the most informative samples for manual labeling; s3: Use semi-supervised learning methods to generate pseudo labels for unlabeled samples using labeled samples; s4: Apply label smoothing technology to reduce the risk of overfitting: y smooth =(1-α)*y+α / K, Among them, α is the smoothing parameter, K is the total number of categories; s5: Retrain the model regularly and update the label generation rules.

8. The data processing system for acquiring tags according to claim 1, characterized in that: The data storage module adopts a distributed storage architecture. The following steps are involved: s1: Use consistent hashing algorithm to shard data: shard=hash(key)mod N, Where N is the number of shards; s2: Implement data replication mechanism to ensure data reliability; s3: Use BloomFilter to build a data index and quickly determine whether the data exists: P falsepositive =(1-e -kn / m ) k , Where k is the number of hash functions, n is the number of inserted elements, and m is the size of the bit array; s4: realize data compression storage, using differential encoding and run-length encoding; s5: Regularly merge and compress data to optimize storage space utilization.

9. The data processing system for acquiring tags according to claim 1, characterized in that: The data retrieval module comprises the following steps: s1: Build an inverted index to support multi-field combination query; s2: Implement similarity search based on LSH: P[h(x)=h(y)]=sim(x,y), Among them, h is the hash function, sim(x, y) is the similarity function; s3: Uses a cache mechanism to speed up access to hot data and adopts an LRU (Least Recently Used) strategy for cache updates; s4: implements distributed query optimization, including query decomposition, parallel execution, and result merging; s5: Provides a RESTful API interface that supports flexible query condition combinations and result filtering.

10. The data processing system for acquiring tags according to claim 1, characterized in that: The visualization module includes the following functions: Data distribution visualization: Use histograms, box plots, etc. to display feature distribution; Feature correlation visualization: Use heatmap to display the correlation coefficient matrix between features; Dimensionality reduction visualization: Use the tSNE (tdistributed Stochastic Neighbor Embedding) algorithm for two-dimensional visualization of high-dimensional data: Decision Boundary Visualization: Plot the decision boundary of a classifier in a two-dimensional feature space; Model performance visualization: Use ROC curves, PR curves, etc. to display model performance; Label evolution visualization: Use Sankey diagram to show the label change trend over time.

Images