Operator processing method and device and related equipment

Abstract

The invention provides an operator processing method and device and related equipment, and relates to the technical field of data processing. The method comprises the steps of obtaining an original data set; inputting the original data set into a plurality of selected data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different types of data processing tasks; based on the plurality of data sets, training to obtain a plurality of data processing models; performing model performance analysis on the plurality of data processing models to obtain a model performance analysis result; and determining an optimal data processing operator combination based on the plurality of data processing operator combinations and the model performance analysis result. According to the method, the optimal data processing operator combination can be effectively screened out to adapt to complex and changeable modeling task requirements, so that the generalization ability and robustness of the model are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an operator processing method, apparatus, and related equipment. Background Art

[0002] In the traditional data processing and model training process, it is usually necessary to perform data preprocessing operations on the original data to form a data set with a unified format and structure for model training.

[0003] Related technologies often rely on empirically defined fixed combinations of operators, such as fixed cleaning strategies, and lack systematic exploration of diverse data processing algorithms and their combinations. This single approach to data processing limits the exploration of diverse data and makes it difficult to adapt to the complex and ever-changing demands of modeling tasks.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The present disclosure provides an operator processing method, apparatus, and related equipment, which can effectively screen out the optimal combination of data processing operators to adapt to complex and changing modeling task requirements, thereby improving the generalization ability and robustness of the model.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to one aspect of the present disclosure, an operator processing method is provided, the method comprising: obtaining an original data set; inputting the original data set into a plurality of selected data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different categories of data processing tasks; training a plurality of data processing models based on the plurality of data sets; performing model performance analysis on the plurality of data processing models to obtain model performance analysis results; and determining an optimal data processing operator combination based on the plurality of data processing operator combinations and the model performance analysis results.

[0008] In some embodiments, model performance analysis is performed on the multiple data processing models to obtain model performance analysis results, including: determining a model performance evaluation operator corresponding to the data processing model based on the data processing model; and performing model performance analysis on the multiple data processing models based on the model performance evaluation operator corresponding to the data processing model to obtain model performance analysis results.

[0009] In some embodiments, determining the optimal data processing operator combination based on the multiple data processing operator combinations and the model performance analysis results includes: encoding each data processing operator combination and its corresponding operator parameters to form a structured parameter vector, and recording the model performance index of the data processing model trained by each data processing operator combination after data processing; establishing a linear relationship model between the model performance index, the data processing operator combination and its corresponding operator parameters based on the structured parameter vectors and model performance index of all data processing operator combinations; and screening out the data processing operator combination and its corresponding operator parameters that achieve the optimal model performance index by analyzing the linear relationship model.

[0010] In some embodiments, the linear relationship model is analyzed to screen out the data processing operator combination and its corresponding operator parameters that optimize the model performance index, including: calculating the contribution weight of each operator combination to the model performance based on the structured parameters corresponding to each data processing operator combination and its performance results on the model performance; sorting all data processing operator combinations according to the contribution weight to obtain the priority of different combinations in improving the model performance; for the adjustable parameters within each data processing operator, determining the impact trend of parameter changes on the model performance through linear analysis, and selecting parameter values ​​that optimize the model performance based on pre-set evaluation indicators; based on the priority and parameter optimization results, screening out the data processing operator combination and its corresponding operator parameters that optimize the model performance.

[0011] In some embodiments, based on the priority and parameter optimization results, the data processing operator combination and its corresponding operator parameters that achieve the best model performance are screened out, including: determining one or more groups of performance evaluation indicators based on the data processing model type; using a weighted combination method to perform a fusion calculation on one or more groups of performance evaluation indicators, and with the goal of making the weighted combination result optimal, screening out the optimal data processing operator combination and its corresponding operator parameters.

[0012] In some embodiments, the data processing task includes at least one of the following: data cleaning, data deduplication, and sensitive content filtering.

[0013] In some embodiments, the data processing model includes at least one of the following: a linear regression model, a logistic regression model, a decision tree, a support vector machine, and a neural network model.

[0014] In some embodiments, the original data set includes at least one of the following: a text data set, an image data set, an audio data set, and a video data set.

[0015] According to another aspect of the present disclosure, there is also an operator processing device, which includes: an acquisition module for acquiring an original data set; a processing module for inputting the original data set into a selected plurality of data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different categories of data processing tasks; a training module for training, based on the plurality of data sets, to obtain a plurality of data processing models; an analysis module for performing model performance analysis on the plurality of data processing models to obtain model performance analysis results; and a determination module for determining the optimal data processing operator combination based on the plurality of data processing operator combinations and the model performance analysis results.

[0016] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned operator processing methods by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements any one of the above-mentioned operator processing methods.

[0018] According to another aspect of the present disclosure, a computer program product is further provided, including: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, any one of the above-mentioned operator processing methods is implemented.

[0019] An operator processing method, apparatus, and related equipment provided in an embodiment of the present disclosure include: obtaining an original data set; inputting the original data set into a selected plurality of data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different categories of data processing tasks; training a plurality of data processing models based on the plurality of data sets; performing model performance analysis on the plurality of data processing models to obtain model performance analysis results; and determining the optimal data processing operator combination based on the plurality of data processing operator combinations and the model performance analysis results. By systematically applying a plurality of data processing operator combinations and combining a quantitative evaluation mechanism based on model performance analysis, the optimal data processing operator combination can be effectively screened out to adapt to the complex and changing modeling task requirements, thereby improving the generalization ability and robustness of the model.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0022] Figure 1 A schematic diagram showing the system architecture of an operator processing method in an embodiment of the present disclosure is shown;

[0023] Figure 2 A schematic flow chart of an operator processing method in an embodiment of the present disclosure is shown;

[0024] Figure 3 A flow chart of an operator processing method in an embodiment of the present disclosure is shown;

[0025] Figure 4 A flow chart for determining an optimal data processing operator combination according to an embodiment of the present disclosure is shown;

[0026] Figure 5 A flowchart of a method for selecting a data processing operator combination and its corresponding operator parameters for achieving an optimal model performance index in an embodiment of the present disclosure is shown;

[0027] Figure 6 A schematic diagram of an operator processing device in an embodiment of the present disclosure is shown;

[0028] Figure 7 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0030] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0032] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the operator processing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0033] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103 , and can be a wired network or a wireless network.

[0034] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0035] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0036] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile phone client, a PC client, etc.

[0037] The server 103 may be a server that provides various services, such as a background management server that provides support for the devices operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0038] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0039] Those skilled in the art will know that Figure 3 The number of terminal devices, networks, and servers in the embodiment is merely illustrative, and any number of terminal devices, networks, and servers may be provided based on actual needs. This embodiment of the present disclosure does not limit this.

[0040] Under the above system architecture, an operator processing method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities.

[0041] In some embodiments, the operator processing method provided in the embodiments of the present disclosure can be executed by the terminal device of the above-mentioned system architecture; in other embodiments, the operator processing method provided in the embodiments of the present disclosure can be executed by the server in the above-mentioned system architecture; in other embodiments, the operator processing method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0042] Figure 2 A schematic flow chart of an operator processing method in an embodiment of the present disclosure is shown. Figure 3 A flow chart of an operator processing method in an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the operator processing method provided in the embodiment of the present disclosure includes the following steps:

[0043] S302, obtaining the original data set.

[0044] In this example, a raw dataset refers to the initial data set without any preprocessing or transformation. It contains all information directly obtained from the data source, which may include noise, errors, missing values, etc. The quality of the raw dataset directly affects the effectiveness of subsequent data processing and model training. Raw data can come from databases, file systems, sensors, or other data sources.

[0045] In some embodiments, the original data set can be divided into a text data set, an image data set, an audio data set, and a video data set. Among them, the text data set is a data set consisting of a series of texts, such as articles, comments, emails, social media posts, etc. The image data set contains a series of image files, such as photos, scans, medical images, etc. The audio data set is a collection of voice recordings, music clips, environmental sounds, etc. The video data set is a dynamic image sequence consisting of a series of continuous frames, which is suitable for video analysis, action recognition and other fields. Video data processing usually includes complex operations such as frame extraction, object tracking, and scene segmentation.

[0046] S304: Input the original data set into the selected multiple data processing operator combinations for processing to obtain multiple data sets, wherein the multiple data processing operator combinations are algorithm combinations in different categories of data processing tasks.

[0047] In this embodiment, a data processing operator combination refers to a series of algorithm combinations designed for specific data types (such as text, images, and audio). These algorithms cover multiple processing tasks, including data cleansing, data deduplication, and sensitive content filtering. Each aspect includes multiple algorithms with different functional utilities. These algorithms are combined to process raw data to improve data quality and usability.

[0048] Data cleaning aims to remove noise, errors, and other irrelevant information from the data. Text data cleaning: For example, regular expression matching is used to remove special characters, garbled characters, HTML tags, and other noise from the text, and the Norvig spelling correction algorithm is used to correct spelling errors. Speech data cleaning: Spectral subtraction is used to remove background noise, and the Wiener filter algorithm is used to enhance speech signals. Image data cleaning: Gaussian filtering is used to smooth images and remove noise, or image inpainting algorithms based on partial differential equations (PDEs) are used to repair damaged areas.

[0049] The goal of data deduplication is to identify and remove duplicate data entries. For text data deduplication, duplicate text can be identified by calculating text similarity based on hash algorithms such as Message-Digest Algorithm 5 (MD5), Secure Hash Algorithm 256-bit (SHA-256), or SimHash. For speech data deduplication, Mel-Frequency Cepstral Coefficients (MFCC) features are extracted and duplicates are determined based on feature vector distance, or the dynamic time warping (DTW) algorithm is used to compare the similarity of speech sequences. For image data deduplication, perceptual hashes (such as Average Hash (AHash), Perceptual Hash (PHash), and Difference Hash (DHash)) are calculated or local feature matching algorithms (such as Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF)) are used to identify duplicate images.

[0050] Sensitive content filtering is used to detect and filter data containing sensitive information. Text sensitive content filtering: For example, this involves building a library of sensitive terms and using string matching algorithms (such as Brute-Force (BF) and Knuth-Morris-Pratt (KMP)), or applying deep learning models (such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variants) to perform more complex semantic analysis. Speech sensitive content filtering: This involves identifying sensitive terms based on Hidden Markov Models (HMM) or other acoustic models. Speech content can also be classified using deep learning architectures combined with attention mechanisms. Image sensitive content filtering: For example, this involves using object detection algorithms (such as Faster Region-based Convolutional Neural Networks (Faster R-CNN) and You Only Look Once (YOLO)) to identify specific prohibited items or sensitive logos, or using convolutional neural network models to perform overall image classification.

[0051] The multiple data processing operator combinations are algorithm combinations in different categories of data processing tasks, that is, each combination will contain a specific algorithm. The following examples illustrate how to process the original data set through different algorithm combinations.

[0052] For example, for text data processing: Combination 1: Regular expression matching + MD5 hash deduplication; Noise removal: Use regular expression matching (such as the re library in Python) to write specific regular expressions to identify and remove HTML tags, special characters, and garbled characters; Data deduplication: Calculate the MD5 hash value of the text. If the hash values ​​of two texts are the same, they are considered to be duplicate texts and deleted. Combination 2: Norvig spelling correction + SimHash deduplication; Spelling correction: Apply the Norvig spelling correction algorithm to find the most likely correct spelling based on the edit distance (such as Levenshtein distance) and the probability of word occurrence in the language model; Data deduplication: Use the SimHash algorithm to convert text features into binary signatures, calculate the Hamming distance between signatures, and if the similarity exceeds a certain threshold, it is considered a duplicate text. Combination method 3: Regular expression matching + Norvig spelling correction + CNN sensitive content filtering; Noise removal: Use regular expression matching to remove HTML tags and special characters; Spelling correction: Apply the Norvig spelling correction algorithm to correct spelling errors; Sensitive content filtering: Use a convolutional neural network (CNN) to preprocess the text, convert it into word vectors, and then input it into the model. The model is trained to classify whether the text contains sensitive content.

[0053] For example, for speech data processing: Combination method 1: spectral subtraction + MFCC feature deduplication; Background noise removal: Use spectral subtraction to estimate the noise spectrum in the frequency domain, subtract the noise spectrum from the noisy speech spectrum to obtain a pure speech spectrum; Data deduplication: Extract the Mel-frequency cepstral coefficient (MFCC) features of the speech, calculate the feature vector distance (such as Euclidean distance or cosine distance) between different speech segments, and if the distance is less than a certain threshold, these speech segments are considered to be repeated. Combination method 2: Wiener filtering + DTW deduplication; Speech enhancement: Use the Wiener filter to estimate the power spectrum of the speech signal and the noise signal, design a filter to filter the noisy speech, and enhance the speech signal; Data deduplication: Use the dynamic time warping (DTW) algorithm to find the optimal time warping path between two speech sequences, calculate the similarity between them, and consider speech with extremely high similarity as repeated. Combination method 3: spectral subtraction + HMM sensitive content filtering; background noise removal: use spectral subtraction to remove background noise; sensitive content filtering: build a hidden Markov model (HMM) for sensitive words, use this model to identify the acoustic features corresponding to sensitive words in the voice stream, and detect speech containing sensitive words in real time.

[0054] For example, for image data processing: Combination 1: Gaussian filtering + AHash deduplication; Image denoising: Use Gaussian filtering to perform convolution operations on the image by constructing a Gaussian kernel, smoothing the image and removing Gaussian noise; Data deduplication: Calculate the average hash (AHash) of the image, compare the Hamming distance between the image hash values, and determine whether the image is duplicated. Combination 2: Criminisi algorithm + SIFT feature matching deduplication; Image restoration: Use the Criminisi algorithm based on partial differential equations (PDEs) to search for similar image blocks in known areas to repair damaged or missing parts; Data deduplication: Use the scale-invariant feature transform (SIFT) to extract local feature points of the image, and use a feature point matching algorithm (such as FLANN matching) to calculate the number of feature matches between different images. If the number of matches exceeds a certain threshold, the image is considered to be duplicated. Combination 3: Gaussian filtering + Faster R-CNN sensitive content filtering; Image denoising: Use Gaussian filtering to remove noise from the image; Sensitive content filtering: Use the deep learning-based object detection algorithm Faster R-CNN to train the model on a dataset containing sensitive targets, identify sensitive targets in the image, and filter them.

[0055] By systematically applying the aforementioned data processing operator combinations, combined with a quantitative evaluation mechanism based on model performance analysis, we can effectively select the optimal data processing operator combination to adapt to the complex and ever-changing modeling tasks, thereby improving the model's generalization and robustness. This process ensures an efficient and accurate data flow from raw data to final model training, enhancing the flexibility and performance of the entire data processing system.

[0056] This embodiment introduces multiple sets of data processing operators to sample and process the original data, breaking through the limitation of related technologies that only use fixed sample data for model training. Through the combined application of multiple operators, more potential information of the original data is mined, providing a richer and more valuable data foundation for model training, and improving the depth and breadth of data processing.

[0057] S306: Based on the multiple data sets, multiple data processing models are trained.

[0058] In this embodiment, multiple data sets refer to a number of data sets with different structures or contents generated after the original data set is input into a plurality of different data processing operator combinations for processing in the preceding steps. These data sets reflect the representation of the original data under different preprocessing strategies. Training refers to the use of machine learning or statistical modeling methods to learn the mapping relationship between input and output from a given data set, thereby constructing a mathematical model that can predict or classify new data. A data processing model refers to a mathematical model obtained by training based on a processed data set and used to complete a specific task (such as prediction, classification, regression, etc.). The model can be a traditional statistical model or a modern machine learning model.

[0059] This embodiment trains multiple different types of data processing models based on multiple datasets processed using different data processing strategies, providing foundational support for subsequent model performance analysis and the selection of optimal data processing operator combinations. This process not only considers the impact of data diversity on model performance but also implements a collaborative optimization mechanism for data processing and model training.

[0060] The following is a description of the principles, application scenarios, advantages and disadvantages of the main model types that can be used for training:

[0061] Linear regression model: Based on the least squares method, it is assumed that there is a linear relationship between the independent variable and the dependent variable. The model parameters are estimated by minimizing the sum of squares of the errors between the predicted values ​​and the actual observed values. Finally, the following formula is established:

[0062] y=β0+β1x1+β2x2+…+β n x n +ε

[0063] Where y is the dependent variable, x1, x2, x n are independent variables, β0, β1, β2, β n is the parameter to be estimated, and ε is the error term.

[0064] This approach is used for forecasting numerical target variables, such as stock price forecasting and macroeconomic indicator forecasting in the financial sector, as well as for modeling experimental results in the natural sciences. Advantages: Simple model structure, high computational efficiency, strong interpretability, and ease of analyzing relationships between variables. Disadvantages: Reliance on the linear assumption of the data; if the actual relationship is nonlinear, the model fit is poor; and sensitivity to outliers may affect model stability.

[0065] Logistic regression model: A type of generalized linear model that maps the results of linear regression to the interval [0, 1] by introducing a logistic function (such as the Sigmoid function) to represent the probability of an event occurring. Its mathematical expression is:

[0066]

[0067] Among them, P(y=1|x) represents the value of the given input feature vector x=(x1,x2,...,x n ), the probability that the output variable y is equal to 1; y is the output variable (target variable), which is a binary variable (0 or 1); x=(x1,x2,...,x n ) is the input feature (independent variable), which is the variable used by the model for prediction; β0 is the intercept term (also called the bias term), which means that when all input features x1, x2, ..., x n When all are 0, the logit value corresponding to the basic probability of outputting 1; β1, β2, ... β n Represents the regression coefficient (ie weight) of the model.

[0068] It is primarily used for binary classification problems, such as determining whether a disease has occurred in medical diagnosis or whether a customer will default in credit assessment. Advantages: Fast computation, strong model interpretability, and suitability for scenarios with high feature dimensions. Disadvantages: Applicable only to binary classification problems, model stability deteriorates when features are highly correlated, and it cannot be directly extended to multi-classification tasks.

[0069] Decision tree models organize the decision process in a tree-like structure. Each internal node corresponds to an attribute test, branches represent test results, and leaf nodes represent the final category or predicted value. Partitioning is typically based on metrics such as information gain and the Gini coefficient, aiming to achieve higher purity of the resulting child nodes.

[0070] Suitable for classification and regression tasks, it is widely used in areas such as customer segmentation, product recommendations, and risk assessment. Advantages: The model structure is intuitive, easy to visualize and understand; it does not require complex data preprocessing and can handle missing values ​​and outliers. Disadvantages: It is prone to overfitting, especially when the sample size is small or there is a lot of noise; the tree structure may be too complex, affecting the model's generalization ability.

[0071] Support Vector Machine (SVM) model: In the case of linear separability, it finds a hyperplane with the largest margin to separate the two types of samples. For the case of linearly inseparable samples, it uses a kernel function to map the data into a high-dimensional space to make it linearly separable, and then finds the optimal classification hyperplane:

[0072] f(x)=w T x+b

[0073] Among them, f(x) represents the predicted value of sample x, that is, the basis for judging the category to which the sample belongs, T is the transposition symbol, and w Trepresents the transpose of vector w, w·w is the normal vector of the hyperplane, which determines the direction of the hyperplane, x represents the sample eigenvector, and b represents the bias term, which is used to adjust the position of the hyperplane.

[0074] It is used in fields such as image recognition, text classification, and bioinformatics, and is particularly suitable for classification problems involving small samples and high-dimensional data. Advantages: It performs well in high-dimensional spaces, has strong generalization capabilities, and is suitable for handling nonlinear classification problems. Disadvantages: It has high computational complexity, making training efficiency low, especially on large datasets; model parameter selection significantly affects performance, making parameter tuning difficult.

[0075] In this example, a dataset processed using various data processing strategies is used as input to train various models, including linear regression, logistic regression, decision trees, and support vector machines, to obtain diverse model outputs. This multi-model training mechanism not only helps evaluate the actual impact of different data processing strategies on model performance, but also provides a solid empirical foundation for subsequent model performance analysis and the selection of optimal data processing operator combinations.

[0076] S308: Perform model effectiveness analysis on multiple data processing models to obtain model effectiveness analysis results.

[0077] In this implementation, multiple data processing models refer to various types of machine learning or statistical modeling models trained separately in the previous steps based on multiple data sets generated after processing by different data processing operator combinations, including but not limited to linear regression models, logistic regression models, decision trees and support vector machines. Model performance analysis refers to the process of comprehensively quantitatively evaluating the predictive performance of each model in a specific task (such as classification or regression) by introducing standardized evaluation indicators. This process aims to measure the generalization ability, stability and accuracy of the model on a given data set. The model performance analysis result refers to the result set obtained after scoring each model according to the selected evaluation indicator, also called the model performance indicator, which is used to reflect the pros and cons of the model performance under different data processing strategies, and provide a basis for the subsequent screening of the optimal data processing operator combination.

[0078] In some embodiments, a model effectiveness evaluation operator corresponding to the data processing model is determined according to the data processing model; and model effectiveness analysis is performed on multiple data processing models based on the model effectiveness evaluation operator corresponding to the data processing model to obtain a model effectiveness analysis result.

[0079] Specifically, according to the different types of model tasks, corresponding model evaluation indicators are selected for evaluation.

[0080] The following is an example description of the main model evaluation metrics:

[0081] Accuracy: It indicates the ratio of correctly predicted samples to the total number of samples in the classification model. It is an important indicator to measure the overall prediction accuracy of the model. The formula is as follows:

[0082]

[0083] Among them: TP (True Positive): the number of samples that are actually positive and predicted as positive by the model; TN (True Negative): the number of samples that are actually negative and predicted as negative by the model; FP (False Positive): the number of samples that are actually negative but predicted as positive by the model; FN (False Negative): the number of samples that are actually positive but predicted as negative by the model.

[0084] For example, in spam identification tasks, accuracy reflects the model's ability to correctly distinguish spam from legitimate emails. It's important to note that while accuracy is a straightforward evaluation metric, it may not truly reflect model performance when the distribution of positive and negative samples is severely imbalanced (for example, in a disease screening, where the number of patients is far smaller than the number of healthy individuals).

[0085] Recall: It indicates the proportion of all actual positive samples that are correctly identified as positive by the model, reflecting the model's ability to detect positive samples. Mathematical expression:

[0086]

[0087] Example application scenario: In medical diagnosis models, recall reflects the model's ability to identify actual patients. A higher recall minimizes missed diagnoses. It's important to note that while recall is suitable for scenarios focused on "finding all positive samples," it ignores false positives (FPs) and is therefore typically used in conjunction with other metrics, such as precision.

[0088] Mean Squared Error (MSE): represents the average of the squares of the differences between the predicted value and the true value, and is mainly used to evaluate the prediction accuracy of the regression model. Mathematical expression:

[0089]

[0090] Where: n: total number of samples; y i : The true value of the i-th sample; The predicted value of the i-th sample.

[0091] Example application scenario: In house price prediction tasks, the smaller the mean squared error, the closer the model's predicted value is to the actual house price, and the better the model performance. Mean squared error is very sensitive to outliers, and large errors are magnified after squaring. Therefore, it is more suitable for tasks with relatively stable data distributions and no obvious outliers.

[0092] Mean Absolute Error (MAE): represents the average of the absolute errors between the predicted value and the true value, and is also a commonly used regression model evaluation indicator. Mathematical expression:

[0093]

[0094] Where: n: total number of samples; y i : The true value of the i-th sample; The predicted value of the i-th sample.

[0095] Application example: In weather forecasting, mean absolute error (MAE) can intuitively reflect the average deviation between predicted and actual temperature values. Compared to mean squared error (MSE), MAE is less sensitive to outliers and therefore more robust in datasets with noise or outliers. However, its gradient is not continuous, making it less convenient than MSE in certain optimization processes.

[0096] This example systematically analyzes the performance of multiple data processing models, leveraging various model evaluation metrics such as accuracy, recall, mean squared error, and mean absolute error to comprehensively measure the performance differences under different data processing strategies. These evaluation results not only provide an objective basis for model optimization but also provide a data basis for subsequently determining the optimal combination of data processing operators, thereby optimizing the entire process from data preprocessing to model training.

[0097] S310 , determining an optimal data processing operator combination based on multiple data processing operator combinations and model performance analysis results.

[0098] In this embodiment, multiple data processing operator combinations refer to a cross-category, multi-parameter combination of specific algorithms selected in the preceding steps according to different data processing tasks (such as data cleaning, data deduplication, sensitive content filtering, etc.). Each combination represents a strategy for preprocessing and converting raw data. The model performance analysis results refer to a series of performance indicator results, such as accuracy, recall, mean square error, mean absolute error, etc., obtained after evaluating multiple data processing models trained based on data sets generated by different data processing combinations. This result reflects the performance of the model under a specific data processing strategy. The optimal data processing operator combination refers to the set of data processing strategies and their specific algorithm configurations that can make the final model performance reach the best state (such as the highest prediction accuracy, the smallest error, the strongest generalization ability, etc.) among all candidate data processing operator combinations.

[0099] In this embodiment, the data processing process and the model training effect are optimized in a feedback linkage manner. That is, by comparing and analyzing the model performance corresponding to different data processing strategies, the data processing solution that is most conducive to improving model performance is selected from a large number of combinations.

[0100] Specific implementation methods include establishing a mapping relationship between data processing strategies and model performance: associating each combination of data processing operators with its corresponding model effectiveness indicator to form a "combination-performance" mapping table or matrix. Based on actual task requirements, one or more weighted objective functions are constructed to quantify the overall performance of each combination.

[0101] For example, for regression tasks, the weighted average of MSE and MAE can be used as the evaluation criterion; for classification tasks, a balanced score of Accuracy and Recall (such as F1Score) can be comprehensively considered. Execution sorting and optimization mechanism: Based on the above evaluation indicators or objective function values, all data processing operator combinations are sorted, and the optimal combination is selected based on the set optimization goals (such as minimum error, maximum accuracy). Support parameter-level optimization and recommendation: Based on the selected optimal combination, the relationship between the parameter change trend of its internal operators and model performance is further analyzed to recommend the optimal parameter settings for each operator.

[0102] Figure 4 A flowchart for determining the optimal data processing operator combination provided by the embodiment of the present disclosure, combined with Figure 4 As shown, the method provided in this embodiment for determining the optimal data processing operator combination based on multiple data processing operator combinations and model performance analysis results may include:

[0103] S402, encoding each data processing operator combination and its corresponding operator parameters to form a structured parameter vector, and recording the model performance index of the data processing model trained by each data processing operator combination after data processing.

[0104] In this embodiment, the data processing operator combination refers to a set of algorithm combinations used to preprocess raw data. For example, in text data processing, it can include noise removal (such as regular expression matching), data deduplication (such as hash value-based deduplication) and sensitive content filtering (such as classification filtering based on deep learning). Operator parameters are specific parameters that can be adjusted within each operator. For example, when using regular expressions to remove noise, the specific pattern string of the regular expression; in hash value-based data deduplication, the selection of hash functions and threshold settings, etc. Structured parameter vector: Each data processing operator combination and its parameters are encoded in a unified format as a numerical vector to facilitate subsequent modeling and analysis. For example, a specific combination may be encoded as [1, 3, 2, 0.01], where each number represents a specific algorithm and its parameter configuration. Model performance indicators are quantitative indicators used to evaluate model performance, such as accuracy, recall, mean square error (MSE), etc. These indicators reflect the performance of the model under a specific data processing strategy.

[0105] For example, assume there are three different text data processing combinations: Combination 1: Use regular expression A to remove noise, the SimHash threshold is 3, the number of CNN layers is 2, and the learning rate is 0.01; Combination 2: Use regular expression B to remove noise, the SimHash threshold is 5, the number of CNN layers is 3, and the learning rate is 0.005; Combination 3: Use regular expression C to remove noise, the SimHash threshold is 2, the number of CNN layers is 2, and the learning rate is 0.02.

[0106] Each of the above combinations and its parameters can be encoded as a structured parameter vector, for example: combination one: [1, 3, 2, 0.01]; combination two: [2, 5, 3, 0.005]; combination three: [3, 2, 2, 0.02].

[0107] Then, each combination was applied to the corresponding data set to train a data processing model and record the performance indicators of each model. For example: combination one: accuracy rate is 0.85; combination two: accuracy rate is 0.87; combination three: accuracy rate is 0.89.

[0108] S404: Based on the structured parameter vectors of all data processing operator combinations and the model performance index, a linear relationship model between the model performance index, the data processing operator combination and its corresponding operator parameters is established.

[0109] In this embodiment, a linear relationship model is a mathematical model that represents an approximate linear relationship between a dependent variable (such as a model performance indicator) and multiple independent variables (such as various operator parameters). This model helps understand the degree to which different parameters affect model performance. The least squares method is a commonly used fitting method used to estimate the parameters in a linear relationship model so that the sum of squared errors between the predicted value and the actual value is minimized. The formula is as follows:

[0110] y=β0+β1x1+β2x2+…+β i x i +∈

[0111] Where y is the dependent variable, x1, x2, x i are independent variables, β0, β1, β2, β i is the parameter to be estimated, and ∈ is the error term. Least squares estimation can be used to obtain predicted values ​​for y and estimated values ​​for βi. By constructing linear models for different data processing operators, different predicted values ​​and estimated results can be obtained. Subsequently, by evaluating the operators based on the model, the optimal operator combination and the optimal operator can be obtained.

[0112] For example, based on the structured parameter vector and model performance index generated in the previous step, the following linear relationship model can be constructed: Assuming that the model performance index is accuracy, the model form can be expressed as:

[0113] Accuracy = β0 + β1 * regularization mode number + β2 * SimHash threshold + β3 * number of CNN layers + β4 * learning rate

[0114] By fitting the above model with the least squares method, the influence weight of each parameter on the model performance (i.e., the β coefficient) can be obtained.

[0115] S406 , by analyzing the linear relationship model, screen out the data processing operator combination and its corresponding operator parameters that optimize the model performance index.

[0116] In this embodiment, model analysis includes significance testing, influence direction analysis, and contribution ranking of linear model parameters to determine which parameters have the greatest impact on model performance. The optimal data processing operator combination refers to the set of data processing strategies that, among all candidate combinations, maximizes model performance indicators (e.g., maximum accuracy or minimum error). Operator parameter optimization, based on the selected optimal combination, further adjusts its internal parameters to maximize performance.

[0117] Continuing with the above example, the following conclusions are drawn through linear model analysis: the regularization pattern parameter (β1) is negatively correlated, indicating that a more complex regularization pattern may actually reduce model performance; the number of CNN layers (β3) is positively correlated, indicating that increasing network depth helps improve accuracy; the SimHash threshold (β2) and learning rate (β4) also have a certain impact on the model.

[0118] Based on this, we can eliminate certain inefficient operator combinations (such as combination 1). Among the remaining combinations, we select the one with the highest overall score (such as combination 3). We then further fine-tune the parameters in combination 3 (for example, trying a learning rate of 0.025) to achieve even better model performance. The optimal combination was ultimately determined to be: Regularization mode = C, SimHash threshold = 2, CNN layers = 2, and learning rate = 0.025; the corresponding model accuracy was 0.90.

[0119] Through this systematic analysis and screening process, we can effectively find the optimal combination of data processing operators and their parameter configuration, thereby improving the effectiveness of the entire data processing process.

[0120] In some embodiments, Figure 5 The present invention provides a flowchart for screening out the combination of data processing operators and their corresponding operator parameters to achieve the optimal model performance index, combined with Figure 5 As shown, the embodiment of the present disclosure analyzes the linear relationship model to screen out the data processing operator combination and its corresponding operator parameters that optimize the model performance index, which may include the following steps:

[0121] S502: Based on the structured parameters corresponding to each data processing operator combination and its performance results on the model effectiveness, the contribution weight of each operator combination to the model performance is calculated.

[0122] In this embodiment, the contribution weight represents the importance of a certain data processing operator combination to improving model performance, and is usually obtained through statistical analysis or machine learning methods.

[0123] Suppose there are three text data processing combinations: Combination A: Regular Expression Cleansing + Hash Deduplication + Sensitive Word Filtering; Combination B: Norvig Spelling Correction + SimHash Deduplication + CNN Sensitive Content Recognition; Combination C: Regular Expression Cleansing + SimHash Deduplication + CNN Sensitive Content Recognition. After training a model for each combination, the accuracy is recorded as follows: Combination A → Accuracy = 0.82; Combination B → Accuracy = 0.85; Combination C → Accuracy = 0.89. Then, combining their structured parameter vectors, a linear model is constructed or regression analysis is performed to calculate the contribution weight of each combination to the model performance improvement. Assume that the results are: Combination A Contribution Weight: 0.15; Combination B Contribution Weight: 0.30; Combination C Contribution Weight: 0.55, indicating that Combination C contributes the most to improving model performance.

[0124] S504: Sort all data processing operator combinations according to their contribution weights to obtain the priorities of different combinations in improving model performance.

[0125] In this example, the contribution weight ranking is based on the contribution value of each combination calculated in the previous step, arranged from high to low, thus forming a priority sequence. Priority is used to measure the importance of different combinations in improving model performance, which helps to select the most effective combination for subsequent optimization.

[0126] Continuing with the above example, the ranking by contribution weight is: combination C (weight 0.55); combination B (weight 0.30); combination A (weight 0.15). From this, it can be judged that combination C has the highest priority and should be given priority for in-depth analysis and parameter optimization.

[0127] S506 , for the adjustable parameters within each data processing operator, determine the impact trend of parameter changes on model performance through linear analysis, and select parameter values ​​that optimize model performance based on pre-set evaluation indicators.

[0128] In this embodiment, adjustable parameters refer to specific numerical values ​​that can be adjusted in the data processing operator, such as the similarity threshold in the SimHash algorithm, the learning rate of the CNN model, the pattern string of the regular expression, etc. Linear analysis: Analyze the relationship between parameter changes and model performance by constructing a linear model to determine the trend of the influence of parameters on the model effect (such as positive correlation, negative correlation). Evaluation indicators: such as accuracy, recall rate, mean square error, etc., are used to measure the performance of the model. The optimal parameter value refers to the parameter configuration that can achieve the best performance of the model in the current combination.

[0129] Taking combination C as an example, it includes the following three operators and their adjustable parameters: regular expression cleaning: pattern strings A / B / C; SimHash deduplication: thresholds 2 / 3 / 4; CNN sensitive content recognition: learning rates 0.01 / 0.02 / 0.03. Linear modeling analysis of these parameters revealed that pattern string C resulted in higher accuracy; a SimHash threshold of 2 resulted in more accurate duplicate recognition; and a learning rate of 0.02 resulted in faster CNN model convergence and higher accuracy. Therefore, the recommended parameter configuration is: regular expression pattern string = C; SimHash threshold = 2; CNN learning rate = 0.02.

[0130] S508 , based on the priority and parameter optimization results, screen out the data processing operator combination and its corresponding operator parameters that optimize the model performance.

[0131] In this embodiment, the parameter optimization result refers to the optimal parameter settings for each combination determined after linear analysis. The optimal data processing operator combination refers to the set of data processing strategies and their specific parameter configurations that achieve the best model performance among all candidate combinations, taking into account the priorities and parameter optimization results.

[0132] In some embodiments, based on the priority and parameter optimization results, the data processing operator combination and its corresponding operator parameters that optimize the model performance are screened out, including: determining one or more groups of performance evaluation indicators based on the data processing model type; using a weighted combination method to perform fusion calculations on one or more groups of performance evaluation indicators, and screening out the optimal data processing operator combination and its corresponding operator parameters with the goal of optimizing the weighted combination result. In this embodiment, when the model training method uses linear analysis, the mean square error and mean absolute error are selected as model evaluation operators. The larger the mean square error and mean absolute error, the worse the model training effect. Therefore, after traversing all operators and operator combinations, the optimal solution is generated from the following optimization problem:

[0133]

[0134] Where c is a hyperparameter that is selected based on the specific problem. Based on the above comprehensive evaluation indicators, the ranking results of operator combinations can also be obtained.

[0135] When the training model method uses a logistic regression model, a decision tree model, or a support vector machine model, accuracy and recall are selected as model evaluation operators. The higher the accuracy and recall, the better the model training effect. Similarly, traversing all operators and operator combinations, the optimal solution is generated by the following optimization problem:

[0136]

[0137] Then the sorting result of the operator combination is obtained.

[0138] The method of this embodiment is not only applicable to different types of data (text, image, audio, etc.), but also can adjust hyperparameters (such as c value) according to specific application scenarios and task requirements, so that the whole process has a high degree of adaptability and flexibility, and can be widely used in various practical problems. Through structured parameter coding, linear relationship modeling and sorting screening based on contribution weights, this method greatly improves the overall efficiency from data preprocessing to model training, reduces the need for manual intervention, and achieves a high degree of automation. Based on the results of quantitative analysis, this method provides scientific decision support, helping to quickly locate the optimal solution among many possible data processing operators. This method framework is not limited to a specific field, and its versatility and scalability enable it to be widely used in multiple fields (such as finance, medical care, education, etc.), promoting the in-depth development of data analysis technology in a wider range of fields.

[0139] Based on the same inventive concept, the present disclosure also provides an operator processing device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0140] Figure 6 A schematic diagram of an operator processing device in an embodiment of the present disclosure is shown. Figure 6 As shown, the device includes: an acquisition module 61, a processing module 62, a training module 63, an analysis module 64 and a determination module 65; the acquisition module 61 is used to acquire an original data set, and the original data set includes at least one of the following: a text data set, an image data set, an audio data set, and a video data set; the processing module 62 is used to input the original data set into a selected plurality of data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different categories of data processing tasks; the training module 63 is used to train and obtain a plurality of data processing models based on the plurality of data sets; the analysis module 64 is used to perform model performance analysis on the plurality of data processing models to obtain model performance analysis results; the determination module 65 is used to determine the optimal data processing operator combination based on the plurality of data processing operator combinations and the model performance analysis results.

[0141] In some embodiments, the analysis module 64 is specifically used to: determine the model performance evaluation operator corresponding to the data processing model based on the data processing model; and perform model performance analysis on the multiple data processing models based on the model performance evaluation operator corresponding to the data processing model to obtain model performance analysis results.

[0142] In some embodiments, the analysis module 64 is specifically used to: encode each data processing operator combination and its corresponding operator parameters to form a structured parameter vector, and record the model performance index of the data processing model trained by each data processing operator combination after data processing; based on the structured parameter vectors and model performance indexes of all data processing operator combinations, establish a linear relationship model between the model performance index, the data processing operator combination and its corresponding operator parameters; by analyzing the linear relationship model, screen out the data processing operator combination and its corresponding operator parameters that achieve the optimal model performance index.

[0143] In some embodiments, the analysis module 64 is specifically used to: calculate the contribution weight of each operator combination to the model performance based on the structured parameters corresponding to each data processing operator combination and its performance results on the model performance; sort all data processing operator combinations according to the contribution weight to obtain the priority of different combinations in improving the model performance; for the adjustable parameters within each data processing operator, determine the impact trend of parameter changes on the model performance through linear analysis, and select the parameter value that optimizes the model performance based on pre-set evaluation indicators; according to the priority and parameter optimization results, screen out the data processing operator combination and its corresponding operator parameters that achieve the best model performance.

[0144] In some embodiments, based on the priority and parameter optimization results, the data processing operator combination and its corresponding operator parameters that achieve the best model performance are screened out, including: determining one or more groups of performance evaluation indicators based on the data processing model type; using a weighted combination method to perform a fusion calculation on one or more groups of performance evaluation indicators, and with the goal of making the weighted combination result optimal, screening out the optimal data processing operator combination and its corresponding operator parameters.

[0145] In some embodiments, the data processing task includes at least one of the following: data cleaning, data deduplication, and sensitive content filtering.

[0146] In some embodiments, the data processing model includes at least one of the following: a linear regression model, a logistic regression model, a decision tree, a support vector machine, and a neural network model.

[0147] In some embodiments, the original data set includes at least one of the following: a text data set, an image data set, an audio data set, and a video data set.

[0148] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned apparatus embodiment are the same as those implemented by the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules, as part of the apparatus, can be executed in a computer system, such as a set of computer-executable instructions.

[0149] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0150] Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-mentioned operator processing methods by executing the executable instructions. Since the principles for solving the problem in this electronic device embodiment are similar to those in the above-mentioned method embodiment, the implementation of this electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0151] Refer to the following Figure 7 700 according to this embodiment of the present disclosure will be described. Figure 7 The electronic device 700 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0152] like Figure 7 As shown, electronic device 700 is implemented as a general-purpose computing device. Components of electronic device 700 may include, but are not limited to, the aforementioned at least one processing unit 710, the aforementioned at least one storage unit 720, and a bus 730 connecting various system components (including storage unit 720 and processing unit 710).

[0153] The storage unit stores a program code, and the program code can be executed by the processing unit 710, so that the processing unit 710 performs the steps of various exemplary embodiments of the present disclosure described in the "Exemplary Method" section of this specification. For example, the processing unit 710 can perform the following steps of the above-mentioned method embodiment: inputting the original data set into a selected plurality of data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different categories of data processing tasks; based on the plurality of data sets, training to obtain a plurality of data processing models; performing model performance analysis on the plurality of data processing models to obtain model performance analysis results; and determining the optimal data processing operator combination based on the plurality of data processing operator combinations and the model performance analysis results.

[0154] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202 , and may further include a read-only memory unit (ROM) 7203 .

[0155] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0156] Bus 730 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0157] The electronic device 700 can also communicate with one or more external devices 740 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 700, and / or any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 750. Furthermore, the electronic device 700 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 700, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0159] Based on the same inventive concept, embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements any of the aforementioned operator processing methods. Because the principles underlying the problem solved by this computer-readable storage medium embodiment are similar to those of the aforementioned method embodiment, the implementation of this computer-readable storage medium embodiment can be referenced to the implementation of the aforementioned method embodiment, and any repetitions will not be repeated.

[0160] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] In the present disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0162] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0163] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0164] Based on the same inventive concept, embodiments of the present disclosure further provide a computer program product, including a computer program or instructions, which, when executed by a processor, implements the operator processing method of any one of the above-mentioned method embodiments. Since the principles for solving the problem in this computer program product embodiment are similar to those in the above-mentioned method embodiment, the implementation of this computer program product embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated here.

[0165] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0166] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0167] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0168] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. An operator processing method, characterized in that: The method comprises: Get the original dataset; Inputting the original data set into a plurality of selected data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different categories of data processing tasks; Based on the multiple data sets, multiple data processing models are trained; Performing model effectiveness analysis on the multiple data processing models to obtain model effectiveness analysis results; An optimal data processing operator combination is determined based on the multiple data processing operator combinations and the model performance analysis result.

2. The operator processing method according to claim 1, characterized in that: Performing model effectiveness analysis on the multiple data processing models to obtain model effectiveness analysis results includes: Determining a model effectiveness evaluation operator corresponding to the data processing model according to the data processing model; Model effectiveness analysis is performed on the multiple data processing models based on the model effectiveness evaluation operators corresponding to the data processing models to obtain model effectiveness analysis results.

3. The operator processing method according to claim 1, characterized in that: The determining of the optimal data processing operator combination based on the multiple data processing operator combinations and the model performance analysis result includes: Encode each data processing operator combination and its corresponding operator parameters to form a structured parameter vector, and record the model performance index of the data processing model trained by each data processing operator combination after data processing; Based on the structured parameter vectors and model performance indicators of all data processing operator combinations, a linear relationship model between the model performance indicators, data processing operator combinations and their corresponding operator parameters is established; By analyzing the linear relationship model, a data processing operator combination and its corresponding operator parameters that can achieve the optimal model performance index are screened out.

4. The operator processing method according to claim 3, characterized in that: The linear relationship model is analyzed to select a data processing operator combination and its corresponding operator parameters that optimizes the model performance index, including: Based on the structural parameters corresponding to each data processing operator combination and its performance on model effectiveness, calculate the contribution weight of each operator combination to model performance; Sorting all data processing operator combinations according to the contribution weights to obtain the priorities of different combinations in improving model performance; For each adjustable parameter within the data processing operator, the impact trend of parameter changes on model performance is determined through linear analysis, and the parameter values ​​that optimize model performance are selected based on pre-set evaluation indicators; According to the priority and parameter optimization results, the data processing operator combination and its corresponding operator parameters that optimize the model performance are screened out.

5. The operator processing method according to claim 4, characterized in that: Based on the priorities and parameter optimization results, the data processing operator combination and its corresponding operator parameters that optimize the model performance are selected, including: Determining one or more groups of performance evaluation indicators according to the data processing model type; A weighted combination method is used to fuse and calculate one or more groups of performance evaluation indicators, and with the goal of optimizing the weighted combination result, the optimal data processing operator combination and its corresponding operator parameters are screened out.

6. The operator processing method according to claim 1, characterized in that: The data processing tasks include at least one of the following: data cleaning, data deduplication and sensitive content filtering.

7. The operator processing method according to claim 1, characterized in that: The data processing model includes at least one of the following: a linear regression model, a logistic regression model, a decision tree, a support vector machine and a neural network model.

8. The operator processing method according to claim 1, characterized in that: The original data set includes at least one of the following: a text data set, an image data set, an audio data set, and a video data set.

9. An operator processing device, characterized in that: The device comprises: Acquisition module, used to obtain the original data set; a processing module, configured to input the original data set into a plurality of selected data processing operator combinations for processing to obtain a plurality of data sets, wherein the plurality of data processing operator combinations are algorithm combinations in different categories of data processing tasks; A training module, configured to train and obtain multiple data processing models based on the multiple data sets; An analysis module, configured to perform model effectiveness analysis on the plurality of data processing models to obtain model effectiveness analysis results; A determination module is used to determine the optimal data processing operator combination based on the multiple data processing operator combinations and the model performance analysis result.

10. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the operator processing method according to any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the operator processing method according to any one of claims 1 to 8.

Images