Data conversion method and device based on machine learning, medium and equipment

Abstract

The invention discloses a data conversion method and device based on machine learning, a medium and equipment, and the method comprises the steps: obtaining original data, and carrying out the preprocessing of the original data; performing feature selection on the preprocessed data based on a feature selection algorithm; inputting the data after feature selection into a corresponding trained rule generation model based on a data type to generate a data conversion rule; wherein the rule generation model is constructed and trained according to different data types; acquiring a real-time data stream, optimizing the rule based on the trained rule optimization model, and / or adaptively adjusting the rule according to the real-time monitoring data; and processing the data through the adjusted rule, and performing anomaly detection and correction on the data after rule processing. According to the method, the automatic rule is generated through real-time data pattern recognition, rule optimization and comprehensive data detection and correction are carried out in a self-adaptive mode, and the efficiency and accuracy of data processing are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and analysis, and in particular to a data conversion method, device, medium and equipment based on machine learning. Background Art

[0002] With the rapid development of information technology, data has become an important asset for modern enterprises and organizations. However, these data often come from different systems, in different formats, and of varying quality, which poses a huge challenge to the effective use of data. Traditional data processing methods rely on manually defined rules, which is not only inefficient, but also difficult to adapt to the rapid changes in data, and easily leads to errors and inconsistencies in data processing.

[0003] The existing technology mainly has the following problems: 1. Dependence on manual rules: Traditional methods rely on manually defined data conversion rules, which makes it difficult to maintain and update the rules; 2. Data quality issues: There are often noise, missing values ​​and outliers in the data, which affect the accuracy of the data; 3. Insufficient pattern recognition capabilities: Existing technologies have limited capabilities in data pattern recognition and cannot effectively mine the potential value in data; 4. Poor real-time performance: Data processing usually cannot be performed in real time, resulting in information lag. Summary of the invention

[0004] Based on the above background, the purpose of the present invention is to provide a data conversion method based on machine learning, which is applied to scenarios such as big data processing, data cleaning, enhancement, pattern recognition, and multi-source data fusion, aiming to solve the limitations of traditional data processing methods and improve the efficiency and accuracy of data conversion.

[0005] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a data conversion method based on machine learning, comprising: Obtaining raw data and preprocessing the raw data; Perform feature selection on the preprocessed data based on the feature selection algorithm; The data after feature selection is input into the corresponding trained rule generation model based on the data type to generate the corresponding data conversion rules; wherein the rule generation model is constructed and trained according to the data type, including an image data rule generation model and a text data rule generation model; Obtain real-time data streams, optimize rules based on trained rule optimization models, and / or adaptively adjust rules by defining rule adjustment functions based on real-time monitoring data; The data is processed through the adjusted rules, and anomalies are detected and corrected for the data processed by the rules.

[0006] In a second aspect, the present invention provides a data conversion device based on machine learning, comprising: The raw data acquisition module is used to acquire the raw data and pre-process the raw data; The feature selection module is used to select features of data through feature selection algorithms and extract feature subsets from the original data; The rule generation module obtains the data after feature selection and generates corresponding data conversion rules; The rule optimization module is used to optimize the rules by defining the reward function based on the reinforcement learning algorithm, and to adaptively adjust the rules according to the real-time monitoring data; The data monitoring and correction module is used to obtain the data after rule processing, and perform anomaly detection and correction on the data after rule processing.

[0007] In a third aspect, the present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned machine learning-based data conversion method.

[0008] In a fourth aspect, the present invention further provides an electronic device comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0009] The data conversion method based on machine learning provided by the present invention has the following beneficial effects: 1. Real-time data pattern recognition and efficient automatic rule generation: With the help of advanced deep learning models, it is possible to continuously monitor and identify potential patterns and trends in dynamic data streams for real-time analysis, ensure the timeliness and accuracy of data processing, and quickly and automatically create data conversion rules that adapt to different data types and application scenarios, ensuring flexible response when processing data from different industries and fields.

[0010] 2. Adaptive rule optimization mechanism: Automatically monitor changes in data features, and timely adjust and optimize data processing rules to improve the applicability and accuracy of the rules. Through continuous learning and feedback, the system can improve itself to ensure that it still maintains efficient processing capabilities under new data conditions.

[0011] 3. Comprehensive anomaly detection and automatic correction: Timely detect errors in the data stream and perform effective automatic corrections to ensure data quality. Specifically, by applying advanced technologies such as natural language processing and generative adversarial networks, the data cleaning process can automatically remove noise, fill missing values, and generate synthetic data, thereby enriching the diversity of the data set and improving the quality of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 A flowchart of a data conversion method based on machine learning provided by an embodiment of the present invention; Figure 2 A schematic diagram of a data conversion device based on machine learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0015] Embodiment 1

[0016] See also Figure 1 The data conversion method based on machine learning provided by the embodiment of the present invention comprises the following steps: S1: obtain the original data and preprocess the original data; S2: Perform feature selection on the preprocessed data based on the feature selection algorithm; S3: The data after feature selection is input into the rule generation model trained for the corresponding type of data based on the data type to generate the corresponding data conversion rules; wherein the rule generation model is constructed and trained separately according to different data types; S4: Obtain real-time data streams, optimize rules based on trained rule optimization models, and / or adaptively adjust rules based on real-time monitoring data; S5: Process the data using the adjusted rules, and perform anomaly detection and correction on the data processed by the rules.

[0017] The step S1 of acquiring data and preprocessing the data specifically includes: S101, obtaining original data from a data source; The data types obtained from the data source include structured data and unstructured data. Structured data refers to data that can be organized in a predefined way, including databases and CSV (Comma-Separated Values) file data, and unstructured data refers to data that does not follow a predefined format, including text data and image data.

[0018] S102, preprocessing the acquired raw data includes data source identification, data metadata extraction, data cleaning, and data format conversion; wherein, The data source identification is used to determine the source of the data, including the time, location, and method of data collection; Metadata extraction is to extract additional information about the original data, such as data characteristics and attributes, to help understand and use the data; Data cleaning, including noise removal, outlier identification, and duplicate data removal, to improve data quality and ensure the accuracy and effectiveness of subsequent data conversion; Format conversion is the process of converting data from one data format to another, such as converting text format to table format, or converting numerical data to a specific statistical data format. When implementing, the data is converted into a data type suitable for analysis and processing based on specific needs.

[0019] In step S2, feature selection is performed on the preprocessed data by using a feature selection algorithm to extract the most relevant and informative feature subset from the original data to reduce computational costs and help understand the data.

[0020] Automated feature selection algorithms can identify important feature variables and reduce the number of features, thereby avoiding model overfitting. Common feature selection algorithms include filter methods, wrapper methods, and embedded methods. Among them, the filter method selects features by evaluating the relationship between features and target variables. Common methods include variance selection method, chi-square test, Pearson correlation coefficient, etc.; the wrapper method evaluates the performance of feature subsets by building models, and cross-validation is usually used to select the best feature set. Common methods include recursive feature elimination (RFE); the embedding method embeds the feature selection process into the model training process. Common algorithms include LASSO, decision tree, etc. The choice of feature selection algorithm depends on the specific application scenario, the characteristics of the data set, and the model used. In practical applications, it is necessary to combine specific application scenarios and domain knowledge to determine the most appropriate feature selection strategy. Feature selection can not only improve the training efficiency and prediction performance of the model, but also help understand the patterns and relationships behind the data.

[0021] In this embodiment, in order to improve the automation of feature engineering, some ready-made libraries, such as FeatureTools and TPOT, can be used to automatically complete the feature selection process. FeatureTools is an open source library that allows users to automatically generate new features through a simple API.

[0022] In step S3, the data after feature selection is input into the rule generation model trained with the corresponding type of data based on the data type to generate rules. Rule generation is to identify patterns and trends from a large amount of data, discover useful information in the data set, including but not limited to classification, clustering, anomaly detection and prediction, and generate corresponding data conversion rules. The generated data conversion rules can help understand the inherent laws in the data, or be directly used in decision support systems, automated processes, etc.

[0023] It should be noted that the rule generation model is constructed and trained according to different data types, and different types of data correspond to different rule generation models.

[0024] For structured data, it can be constructed based on machine learning algorithms such as decision trees and random forests. The above algorithms can learn the relationship between data features and target variables through training data, thereby generating corresponding rules. That is, by analyzing the data set, effective conversion rules are extracted to ensure flexibility when processing data from different industries and fields. Among them, the decision tree algorithm is a simple and effective classification and regression model that can divide data into different categories through a series of rules. The rules it generates are not only easy to understand, but also can effectively handle nonlinear relationships. Common decision tree algorithms include ID3, C4.5, and CART. The construction process of a decision tree is mainly as follows: (1) Select features: use indicators such as information gain, gain rate, or Gini coefficient to select the best segmentation features; (2) Create nodes: divide nodes according to the selected features to form a tree structure; (3) Recursive segmentation: repeat the above process for each subset until the stopping condition is met (such as reaching the maximum depth or the number of nodes is less than the threshold). The random forest algorithm is an ensemble learning method based on decision trees. It improves the accuracy and stability of the model by constructing multiple decision trees and combining their results. This method can effectively reduce overfitting and improve the model's ability to generalize to new data. The construction process of random forest is as follows: (1) Bootstrap sampling: randomly extract a number of samples from the original data set to form multiple different training sets; (2) Construct a decision tree: construct a decision tree for each training set and randomly select features for segmentation; (3) Voting mechanism: vote on the output results of all trees to determine the final classification result or regression value.

[0025] For text data, since the recurrent neural network (RNN) can effectively process sequence data, model contextual relationships, and dynamic input length, the network structure can be used to build a rule generation model for text data types. The process of building a text data rule generation model based on the recurrent neural network (RNN) includes: 1. Data preparation Data collection: Collect text data that meets the requirements and ensure the diversity and representativeness of the data set; Data preprocessing: including text cleaning (removing impurities, such as punctuation, special symbols, etc.), word segmentation, stem extraction, etc.; Encoding: Convert text data into numerical form. Common methods include bag-of-words model, TF-IDF, Word2Vec, GloVe, etc. For RNN, it is usually chosen to convert words into dense vector representations.

[0026] 2. Build the model Choose a recurrent neural network (RNN) structure. You can choose a basic RNN, a long short-term memory network (LSTM), or a gated recurrent unit (GRU). LSTM and GRU are usually more effective in capturing long-distance dependencies. Define input and output: The input is usually a sequence (such as a sentence or a piece of text), and the output can be the prediction of the next word or the generation of the entire sentence.

[0027] 3. Model training Select an appropriate loss function to measure the gap between the model prediction and the true value. The loss function can be the cross entropy loss function; and combine it with an optimization algorithm (such as Adam, SGD, etc.) to update the model parameters. Divide the prepared data set into a training set and a validation set, train the RNN model through multiple rounds of iterations, and monitor the loss function to avoid overfitting.

[0028] 4. Model Evaluation Use the validation set to evaluate the performance of the model. Common indicators include accuracy, F1-score, perplexity, etc. And adjust the model's hyperparameters (such as learning rate, hidden layer size, batch size, etc.) based on the evaluation results to improve model performance.

[0029] Through the above steps, a rule generation model can be constructed that can extract features from text data and perform classification or recognition. Different generation strategies, such as greedy search, beam search, or temperature sampling, are used to generate rules for text data.

[0030] For image data, since convolutional neural networks (CNNs) are suitable for processing images and time series data and can automatically extract spatial features, this network structure can be used to build a rule generation model for image data types. Furthermore, the process of building a rule generation model for image data includes: 1. Obtain historical data and determine the model structure based on the data type; Obtain historical data, and divide the data into a training set and a validation set according to the time window or the amount of data; determine that the model is constructed using a CNN model structure according to the data type; the CNN model generally includes basic structural units such as a convolutional layer, a pooling layer, and a fully connected layer. The model structure specifically constructed in this embodiment includes a convolutional layer, a pooling layer, and a fully connected layer; wherein the input of the model is the image data after the shape is adjusted, and the output is the probability score of each category. The output result of the model can be used to determine the category with the highest probability of the predicted category in the classification task as the prediction result.

[0031] 2. Input features and labels. The input features of the model are the original image data, and the labels are the corresponding category labels.

[0032] This includes adjusting the shape of the original image: For the CNN model, the input data is usually a four-dimensional array, and its shape is expressed as (batch_size, height, width, channels), where batch_size represents the number of samples input to the model in one training, height represents the height of the image, width represents the width of the image, and channels represents the number of channels of the image, which is 1 for grayscale images and 3 for color images (RGB). In this embodiment, in order to meet the needs of the CNN model, the original image data is adjusted to a shape suitable for the input of the CNN model, that is, (-1, 28, 28, 1), where -1 means that the size of the dimension is automatically calculated so that the number of elements of all channels is consistent; the first 28 means that the height of the image is 28 pixels; the second 28 means that the width of the image is 28 pixels; and 1 means that the number of channels of the image is 1, that is, a grayscale image.

[0033] Label one-hot encoding: Convert the category label to one-hot encoding so that the model can distinguish different categories. In the above example, the label is converted into a binary vector of length 10, where only one element is 1 and the rest are 0, indicating different categories.

[0034] 3. Train the model. Use the training set data to train the model. Combine the cross-entropy loss function and optimization algorithm (such as Adam, SGD) to optimize the model. The cross-entropy loss function is a commonly used loss function for classification problems, which is used to measure the gap between the predicted value and the true value. The optimization algorithm is used to update the parameters of the model to minimize the loss function. Among them, the loss function formula is in the form of a cross-entropy loss function, which is used to measure the gap between the predicted value and the true value. For binary classification problems, the commonly used cross-entropy loss function is the softmax loss function, but for multi-classification problems, the cross-entropy loss function is usually used to measure the gap between the probability distribution of the predicted category and the true category.

[0035] 4. Validation set hyperparameter adjustment: Avoid overfitting by adjusting hyperparameters (such as learning rate, batch size, etc.) on the validation set. The validation set is a dataset different from the training set, which is used to evaluate the performance of the model during the training process and adjust the hyperparameters to optimize the model.

[0036] Through the above steps, a rule generation model can be constructed that can extract features from data and perform classification or identification, thereby generating data conversion rules.

[0037] Furthermore, this step may also include storing the generated rules to construct a rule base, which can support subsequent reuse and optimization and has functions such as dynamic update, version control and rule query.

[0038] The structural design of the rule base usually includes rule ID, rule content, creation time, version information and usage frequency. The rule ID is the ID that uniquely identifies the rule; the rule content is the specific rule description, including conditions and results; the creation time is the time when the rule is generated; the version information is the version number of the rule for version management; the usage frequency is the number of times the rule is called to evaluate the effectiveness of the rule.

[0039] When building a rule base, you can choose a relational database (such as MySQL, PostgreSQL) or a non-relational database (such as MongoDB) to store rule data. Use caching tools such as Redis to improve the efficiency of rule queries. And build a RESTful API interface through Flask or FastAPI to facilitate other systems to call the rule base.

[0040] In this embodiment, the Flask framework and SQLite database are used to build a rule base, which specifically includes defining a database model of the rule base, defining rule creation and query interfaces, creating rule addition and query interfaces, etc., so as to realize creation, query and deletion of rules.

[0041] S4, obtains real-time data streams, optimizes rules based on the trained rule optimization model, and adaptively adjusts rules based on real-time monitoring data; specifically, S401: Acquire real-time data information to monitor data flow and obtain changing data features; By using Kafka to receive real-time data streams, and processing and analyzing real-time data streams based on Apache Flink or Spark Streaming, data stream characteristics are obtained, and feature distribution is regularly analyzed to identify potential pattern changes. The monitored data features provide data support for subsequent adjustment of rule priorities or scope of application.

[0042] In this embodiment, the real-time data stream is processed based on Apache Flink, including the definition of data feature extraction logic. KeyedProcessFunction can be used to monitor changes in data features. KeyedProcessFunction provides semantics of processing time and event time, which can be used to implement complex rule adjustment logic.

[0043] S402: Based on the acquired real-time data, a rule optimization model is constructed through a reinforcement learning algorithm, and the rules are optimized based on historical data and real-time data streams to ensure the effectiveness of the rules.

[0044] Reinforcement learning is a method of learning how to make optimal decisions through trial and error. It learns how to adapt to the environment through rewards or penalties. Reinforcement learning algorithms include Q-learning, deep Q-network (DQN), and policy gradient methods (such as REINFORCE); Q-learning is a value-based reinforcement learning algorithm suitable for discrete state and action spaces; deep Q-network (DQN) is Q-learning combined with deep learning, suitable for processing complex state spaces; policy gradient methods (such as REINFORCE) are direct optimization strategies suitable for continuous action spaces.

[0045] In specific implementation, it can be implemented based on the OpenAI Gym or RLlib (Ray) environment framework. OpenAI Gym provides a standardized interface for the reinforcement learning environment; RLlib (Ray) is a library for distributed reinforcement learning that supports multiple algorithms.

[0046] In this embodiment, the specific steps of implementing automatic optimization rules based on the Q-learning algorithm include: Define state space: define the current state of the rule, including data characteristics, processing effects, etc.

[0047] Define the action space: Define the actions that can be performed, such as adjusting the parameters of a rule, its priority, or adding new rules.

[0048] Define a reward function: Give rewards based on the degree of improvement in data processing after the rule is applied, encouraging the system to optimize the rules. Ensure that the algorithm receives higher rewards when performance improves, thereby learning more effective rules.

[0049] Among them, the reward function is defined as follows: Q(s, a) ← Q(s, a) + α * (r + γ * π - Q(s, a)) Among them: s is the current state, a is the currently selected action, α is the learning rate, r is the current reward, γ is the discount factor, which determines the importance of future rewards; π is the maximum expected return after the current state is transferred, that is, the maximum possible action value of the maximum expected return of the next state.

[0050] The reward function is used to update the value of the Q table based on the current state, action, reward, and the maximum expected return of the next state. The Q table maps various states of data processing (such as data features, rule validity, etc.). In this way, the Q-learning algorithm can gradually optimize the Q table. Based on the information in the Q table, the optimal strategy can be extracted, and the rules can be automatically adjusted and optimized to achieve better performance or adaptability, so that the selected action can obtain a higher expected return, thereby optimizing the entire decision-making process.

[0051] S403: Based on the real-time information flow obtained in step S401, the rules are adaptively adjusted by defining a rule adjustment function. Specifically, S4031: Monitor the status and performance of the system through Prometheus; and visualize data through Grafana to display the effects of rules and system status in real time, helping to identify areas that need adjustment.

[0052] S4032: Customize the rule adjustment function to adaptively adjust the rules according to the changes in the monitored data characteristics. This includes: By monitoring the changes in the features of the input data in real time, it can be determined whether the rules need to be adjusted. For example, if the value of a feature suddenly changes significantly, it may indicate that the importance of the feature for the classification task has changed, and the corresponding rules may need to be adjusted. Adjusting the corresponding rules includes dynamically adjusting their priorities based on the effects of the rules. For example, if a rule performs well in practice and is critical to the current task, its priority can be increased; conversely, if a rule performs poorly in practice or does not match the current task, its priority can be lowered.

[0053] It should be noted that the above process of dynamically adjusting the priority of rules or other rule adjustment logic according to feature change logic can be achieved by defining a rule adjustment function, which updates the weight or other parameters of the rule according to feature changes.

[0054] In step S5, the data is processed using the adjusted rules, and anomaly detection and correction are performed on the rule-processed data.

[0055] S501: Processing the data through the adjusted rules, obtaining the data after the rule processing, building an anomaly detection model, and performing anomaly detection on the data; including: Build a streaming data processing framework, obtain rule-processed data based on Apache Kafka, and use the basic statistical characteristics of the data (such as mean, variance, distribution characteristics, etc.) to detect outliers. Common methods include Z-score, box plots, etc. Or build an anomaly detection model based on machine learning algorithms, including clustering algorithms, isolation forests, support vector machines (SVM), etc. Among them, clustering algorithms such as K-means or DBSCAN divide data into different clusters to identify anomalies that are far away from other data points; Isolation Forest is an algorithm based on random trees that can effectively identify abnormal data in low-density areas; Support Vector Machines (SVM) separate normal data from abnormal data by finding the optimal hyperplane. It should be noted that building an anomaly detection model to detect data anomalies belongs to the existing technology and will not be repeated here.

[0056] S502: Correct the detected abnormal data; common corrections include data filling, data transformation, and outlier deletion. Data filling includes two methods: mean / median filling and prediction filling. Mean / median filling fills missing values ​​and is suitable for numerical data. Prediction filling can also use regression models or K-nearest neighbor algorithms to predict missing values, which can better maintain the original distribution of the data.

[0057] Data transformation includes standardization and normalization. Standardization is to convert data into a distribution with a mean of 0 and a standard deviation of 1. It is suitable for algorithms that need to meet the normal distribution assumption. Normalization is to scale the data to a specific interval (such as [0, 1]). It is suitable for situations where data of different dimensions need to be compared.

[0058] Deleting outliers means considering directly deleting extreme outliers to avoid negative impact on model training and analysis results.

[0059] S503: Clean and augment the data after anomaly detection and correction; Furthermore, in tasks such as image classification and text generation, especially in natural language processing (NLP) projects, data cleaning and data augmentation through generative adversarial networks (GANs) can also be used to improve data quality.

[0060] Text data cleaning includes noise removal, normalization, stop word removal, and word vectorization. Among them, noise removal is to use regular expressions or NLP tools (such as NLTK, spaCy) to remove special characters, HTML tags, and extra spaces in the text; normalization is to unify the text format (such as case conversion, lemmatization) to ensure consistency; stop word removal is to remove meaningless words (such as "de", "shi", "zai", etc.) through a stop word list; word vectorization is to convert the text into vector form through technologies such as Word2Vec, GloVe, or BERT for subsequent processing.

[0061] Data augmentation is based on generative adversarial networks (GANs). A generative adversarial network is a deep learning model composed of two core parts: a generator and a discriminator. The generator is used to synthesize data, and the discriminator is used to evaluate the authenticity of the synthesized data. The two are trained against each other. Among them, the generative adversarial network (GAN) generates data according to specific conditions (such as class labels) to ensure the diversity and pertinence of the generated data. For different data types, different data augmentation strategies can be adopted. Specifically, Image data: Increase the diversity of image data through techniques such as rotation, translation, scaling, and adding noise; Text data: Generate new text instances by methods such as synonym replacement, random insertion / deletion of words, and back translation (translating the text into another language and then back into the original language) for text data; Time series data: Apply methods such as time series decomposition and perturbation to generate new time series data.

[0062] The machine learning-based data conversion method provided in this embodiment generates automated rules through real-time data pattern recognition, adaptively optimizes the rules, and comprehensively detects and corrects data, significantly improving the efficiency and accuracy of data processing.

[0063] Embodiment Two

[0064] See Figure 2 , this embodiment provides a machine learning-based data conversion device for implementing the machine learning-based data conversion method provided in Embodiment One, including, A raw data acquisition module for acquiring raw data and preprocessing the raw data; The feature selection module is used to select features of data through feature selection algorithms to extract the most relevant and informative feature subsets from the original data; The rule generation module is used to obtain the data after feature selection and generate corresponding data conversion rules; A rule optimization module is used to obtain real-time data streams, optimize rules based on trained rule optimization models, and / or adaptively adjust rules by defining rule adjustment functions according to real-time monitoring data; The data monitoring and correction module is used to obtain the data after rule processing, and perform anomaly detection and correction on the data after rule processing.

[0065] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0066] Embodiment 3

[0067] The present embodiment provides a non-transitory computer-readable storage medium, which stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded by a processor and executes the machine learning-based data conversion method in Embodiment 1.

[0068] Embodiment 4

[0069] This embodiment provides an electronic device, which includes a processor and the non-transitory computer-readable storage medium in Embodiment 3 of the present invention.

[0070] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0072] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

Claims

1. A data conversion method based on machine learning, characterized in that: include, Obtaining raw data and preprocessing the raw data; Perform feature selection on the preprocessed data based on the feature selection algorithm; The data after feature selection is input into the corresponding trained rule generation model based on the data type to generate the corresponding data conversion rules; wherein the rule generation model is constructed and trained according to the data type, including an image data rule generation model and a text data rule generation model; Obtain real-time data streams, optimize rules based on trained rule optimization models, and / or adaptively adjust rules by defining rule adjustment functions based on real-time monitoring data; The data is processed through the adjusted rules, and anomalies are detected and corrected for the data processed by the rules.

2. The data conversion method based on machine learning according to claim 1, characterized in that: The data preprocessing includes data source identification, data metadata extraction, data cleaning and / or data format conversion.

3. The data conversion method based on machine learning according to claim 1, characterized in that: The image data rule generation model is constructed based on a convolutional neural network model, and the text data rule generation model is constructed based on a recurrent neural network model.

4. The data conversion method based on machine learning according to claim 3 is characterized in that: The process of constructing the image data rule generation model based on the convolutional neural network model includes: Obtain image history data and divide the data into training set and validation set; Determine the convolutional neural network (CNN) model structure, including convolutional layers, pooling layers, and fully connected layers; Determine that the input feature of the model is the original image data, and adjust the shape of the original image data; determine that the label of the data is the category label, and perform one-hot encoding conversion on the category label; Use the training set data to train the model and optimize the model in combination with the loss function to update the model parameters and minimize the loss function; Evaluate the performance of the model on the validation set and adjust hyperparameters to optimize the model, including learning rate and / or batch size.

5. The data conversion method based on machine learning according to claim 3 is characterized in that: The process of constructing the text data rule generation model based on the recurrent neural network model includes: Obtain text history data and divide the data into training set and validation set; Determine the RNN model structure, including the basic RNN, LSTM or GRU; Define the input and output of the model. The input is a text sequence, and the output is the prediction of the next word or the generation of the entire sentence. Use the training set data to train the model, combine the loss function to measure the gap between the model prediction and the true value, and update the model parameters through the optimization algorithm; Use the validation set to evaluate the performance of the model and adjust the model's hyperparameters, including learning rate, hidden layer size, and / or batch size, based on the evaluation results.

6. The data conversion method based on machine learning according to claim 1, characterized in that: The construction process of the rule optimization model includes: Define the state space: define the current state of the rule, including data characteristics and / or processing effects; Define the action space: Define the actions that can be performed, including adjusting the parameters and priorities of rules or adding new rules; Define a reward function: reward the system based on the improvement in data processing after the rule is applied, encourage the system to optimize the rules, and ensure that the algorithm receives higher rewards when performance improves, thereby learning more effective rules. Among them, the reward function is defined as: Q(s, a) ← Q(s, a) + α * (r + γ * π - Q(s, a)) Among them: s is the current state, a is the currently selected action, α is the learning rate, r is the current reward, γ is the discount factor, which determines the importance of future rewards; π is the maximum expected return after the current state is transferred, that is, the maximum possible action value of the maximum expected return of the next state.

7. The data conversion method based on machine learning according to claim 1, characterized in that: Adaptive adjustment of rules based on real-time monitoring data includes: Monitor data feature changes in real time, record feature distribution changes and display them visually; Receive real-time data streams to provide real-time data input for rule adjustment; Process and analyze real-time data streams, verify data integrity, and obtain data stream characteristics; According to the changes in the monitored data characteristics, the custom rule adjustment function makes adaptive adjustments to the rules.

8. A data conversion device based on machine learning, characterized in that: The device is used to implement the data conversion method based on machine learning as claimed in claim 1, comprising: The raw data acquisition module is used to acquire the raw data and pre-process the raw data; The feature selection module is used to select features of data through feature selection algorithms and extract feature subsets from the original data; The rule generation module obtains the data after feature selection and generates corresponding data conversion rules; The rule optimization module is used to optimize the rules by defining the reward function based on the reinforcement learning algorithm, and to adaptively adjust the rules according to the real-time monitoring data; The data monitoring and correction module is used to obtain the data after rule processing, and perform anomaly detection and correction on the data after rule processing.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the machine learning-based data conversion method as described in any one of claims 1-7.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images