One-click table building and model processing task generation method based on deep learning

Abstract

The present invention discloses a one-key table building and model processing task generation method based on deep learning, including S1, using deep learning algorithm to automatically analyze structured and unstructured input data; S2, data relationship intelligent recognition system strengthened by graph neural network; S3, using integrated multimodal data fusion framework to process and integrate information from different data sources; S4, introducing automatic data cleaning and preprocessing system based on deep reinforcement learning; S5, optimizing model performance through genetic algorithm, automatically adjusting the network structure and hyperparameters of the model; S6, implementing dynamic scheduling algorithm based on task priority and resource utilization efficiency, optimizing computing resource allocation in real time, reducing processing delay, and improving system response speed and task processing capability. The present invention has the advantages of high automation, fast response speed, strong adaptability and high resource utilization efficiency.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a one-click table building and model processing task generation method based on deep learning. Background Art

[0002] With the widespread application of big data and cloud computing technologies, automation of data management and processing has become the key to improving productivity and decision-making efficiency. Traditional database establishment and data processing methods rely on manual operations, which are not only time-consuming and labor-intensive, but also prone to errors and inefficiencies when faced with large-scale and complex data. In particular, in terms of data structure identification, database table establishment, and data preprocessing, traditional methods usually lack flexibility and adaptability, and are difficult to meet the needs of modern data-intensive applications.

[0003] In the prior art, database management systems (DBMS) and data processing tools mostly perform tasks based on preset rules and procedures. These tools have obvious defects in the following aspects:

[0004] 1. Lack of flexibility and adaptability: Traditional tools usually cannot automatically identify the type and structure of data, requiring database administrators (DBAs) to manually perform data modeling and table structure design, which not only increases the workload but also increases the probability of errors.

[0005] 2. Inefficiency: In a big data environment, manual processing of data table creation, model selection, and optimization tasks becomes extremely arduous and inefficient, especially when data formats and sources change frequently, traditional methods are difficult to respond quickly.

[0006] 3. Insufficient data preprocessing: Data cleaning and preprocessing are crucial steps in data analysis. Traditional methods often rely on fixed scripts and programs, lack the ability to dynamically adapt to data changes, and cannot effectively handle noise and outliers in the data.

[0007] 4. Uneven task scheduling and resource utilization: When dealing with multiple tasks and large-scale data processing, traditional systems often cannot effectively allocate computing resources, resulting in resource waste or task delays, and cannot ensure the timely processing of high-priority tasks.

[0008] Therefore, how to provide a one-click table building and model processing task generation method based on deep learning is an urgent problem that technicians in this field need to solve. Summary of the invention

[0009] One object of the present invention is to propose a one-click table creation and model processing task generation method based on deep learning. The present invention makes full use of deep learning, natural language processing, genetic algorithm and dynamic task scheduling technology, and describes in detail the process of automating the generation of database tables, data preprocessing, model selection and optimization, and task scheduling. It has the advantages of high degree of automation, fast response speed, strong adaptability and high resource utilization efficiency.

[0010] According to an embodiment of the present invention, a one-click table building and model processing task generation method based on deep learning is characterized by comprising the following steps:

[0011] S1. Use deep learning algorithms to automatically analyze structured and unstructured input data and extract the structure and type of data from the user's natural language instructions;

[0012] S2. The data relationship intelligent recognition system enhanced by the graph neural network automatically builds and optimizes the database table structure based on the analysis and extraction results, and performs intelligent matching and configuration of data fields and data relationships;

[0013] S3, using an integrated multimodal data fusion framework to process and integrate information from different data sources;

[0014] S4. Introduce an automatic data cleaning and preprocessing system based on deep reinforcement learning to automatically select and apply the optimal cleaning and preprocessing strategy based on data type and quality;

[0015] S5. Optimize model performance through genetic algorithms, use variational autoencoders to process data characteristics, and automatically adjust the model's network structure and hyperparameters;

[0016] S6. Based on task priority and resource utilization efficiency, implement dynamic scheduling algorithms to optimize computing resource allocation in real time, reduce processing delays, and improve system response speed and task processing capabilities.

[0017] Optionally, the S1 specifically includes:

[0018] S11. The system first receives input including structured and unstructured data. The system receives instructions provided by the user in natural language, indicating the requirements for data processing and table creation;

[0019] S12. For unstructured image data, use convolutional neural network to extract features:

[0020]

[0021] Among them, F(x,y) represents the feature output of the image at position (x,y), K represents the convolution kernel, I represents the input image data, i and j are the index positions in the convolution kernel, and k is the size of the convolution kernel;

[0022] S13. For text data, the system uses long short-term memory network technology for semantic analysis and structural parsing:

[0023] h t =σ)W ih ·x t +b ih +W hh ·h (t-1) +b hh );

[0024] Among them, h t is the hidden state at time step t, x t is the input vector at time step t, W ih and W hh are the weight matrices from input to hidden and hidden to hidden, respectively, and b ih and b hh is the bias term, σ is the activation function;

[0025] S14. Use the support vector machine decision model to combine the features extracted from the image through the convolutional neural network and the features processed from the text through the long short-term memory network to determine the data structure and type:

[0026] f(x)=sgn(w T φ(x)+b);

[0027] Among them, x represents the feature vector, which is composed of the features of image and text data, φ(x) is the function that maps the feature vector to a high-dimensional space, w is the weight vector, b is the bias term, and sgn is the sign function used to determine which category the data belongs to;

[0028] S15. Based on the data structure and type determined by the support vector machine model, the system automatically maps to the corresponding database model, selects a suitable database table structure for each data type, and configures specific data field types. The system automatically generates SQL statements and non-SQL database commands based on the output of the integrated model:

[0029] CREATETABLEIFNOTEXISTStable_name(column_namedata_type,…).

[0030] Optionally, the S2 specifically includes:

[0031] S21. Use graph neural networks to process input data and identify complex relationships and dependencies between data elements:

[0032]

[0033] in, represents the feature vector of node v at time step t, N(v) represents the set of neighbor nodes of node v, c vu is the contribution weight of node u to node v, W (t) is the weight matrix at time step t, is the input feature of node v at time step t, and f is the activation function;

[0034] S22. Automatically identify key entities and their relationships in the data by analyzing the output of the graph neural network:

[0035] r vu =softmax(W r ·concat(h v ,h u ));

[0036] Among them, r vu represents the relationship type from entity v to entity u, W r is the weight matrix for relationship classification, concat(h v ,h u ) is the result of concatenating the feature vectors of entity v and entity u. The softmax function is used to normalize the output into a probability distribution, indicating the possibility of different types of relationships.

[0037] S23. Automatically construct the database table structure based on the data relationships and entity features obtained from the graph neural network, and apply optimization algorithms to select the best table structure and data field type:

[0038]

[0039] Among them, S represents the selected storage structure, T represents the table configuration, V is the set of data entities, E is the set of relationships between entities, and S v is the storage structure of entity v, D v is the data type of entity v, L is the loss function for the mismatch between storage structure and data type, T vu is the table configuration between entities v and u, R vu is the type of relationship between entities, C is the cost function for mismatch between configuration and relationship type, λ1 and λ2 are weight parameters for adjusting the impact of the two losses;

[0040] S24. The system automatically optimizes the index and storage configuration of the database table based on the output of the graph neural network model to improve query efficiency and data processing speed, evaluate the performance of different index strategies, and select the strategy that best suits the current data structure:

[0041] m I (α@Cost storage (I)+β@Cost query (I));

[0042] Among them, I represents the index configuration set, Cost storage (I) is the storage cost under a given index configuration, Cost query (I) is the query cost under the index configuration, α and β are weight parameters to balance the impact of storage and query costs;

[0043] S25. Automatically generate SQL statements and corresponding commands applicable to non-SQL databases, create and configure database tables, and convert optimization results into specific database commands:

[0044]

[0045] Among them, table_name is the table name determined according to the output of the graph neural network, n is the number of fields, column_name i is the name of the i-th field, data_type i Is the data type of the i-th field.

[0046] Optionally, the S3 specifically includes:

[0047] S31. Use an integrated multimodal data fusion framework to process and integrate information from different data sources, including text, image, sound, and video data:

[0048] F integrated =concat(F text ,F image ,F audio ,F video );

[0049] Among them, F integrated represents the fused feature vector, F text , F image , F audio , and F video They represent feature vectors extracted from text, image, audio and video data respectively, and concat represents the concatenation function of feature vectors;

[0050] S32. Analyze the fused feature vectors, identify and extract key instructions for data processing and table building, and use variational autoencoders to further extract features and reduce dimensions:

[0051]

[0052] Where z represents the potential feature space, μ and σ are the encoded mean and variance, ∈ is the noise sampled from the standard normal distribution, Represents element-wise multiplication;

[0053] S33, using the extracted feature vectors to automatically generate task instructions for data processing and table building, using a multi-layer perceptron for feature analysis and task generation:

[0054] y=softmax(W2@σ(W1@x+b1)+b2);

[0055] Among them, x represents the comprehensive feature vector of the input, W1 and W2 are the weight matrices of the network layer, b1 and b2 are bias vectors, σ represents the activation function, and the softmax function is used to convert the network output into a probability distribution, which represents the possibility of table creation and data processing tasks in different databases.

[0056] Optionally, the S4 specifically includes:

[0057] S41. Use an algorithm based on deep reinforcement learning to automatically analyze the type and quality of input data, use state representation learning to encode the data set, and convert the state s of the data into a low-dimensional feature representation through an encoding function φ:

[0058] s ′ =φ(s);

[0059] Among them, s represents the original data state, s ′ Indicates the encoded state;

[0060] S42. According to the encoded data state, use the deep reinforcement learning model to select the most suitable data cleaning and preprocessing strategy, and maximize the long-term reward by optimizing the decision strategy π:

[0061]

[0062] Among them, π * represents the optimal strategy, R t is the reward obtained at time t, γ is the discount factor, which represents the current value of future rewards;

[0063] S43. Apply the selected strategy to perform data cleaning and preprocessing tasks, using statistical analysis-based methods to automatically identify and remove outliers in the data:

[0064]

[0065] Where X is the observed value, μ is the mean, and σ is the standard deviation;

[0066] Choose an appropriate filling strategy based on the type and distribution of the data. For numerical data, use mean filling:

[0067] X filled =X missing ∪μ;

[0068] Among them, X missing is the missing data point, μ is the mean of the non-missing data points, and ∪ represents the replacement operation;

[0069] Standardize the data to eliminate dimension effects and improve algorithm performance:

[0070]

[0071] Where X is the original data, min(X) and max(X) are the minimum and maximum values ​​of the data respectively;

[0072] S44. Continuously monitor the effect of data preprocessing and adjust the strategy according to the actual effect. Use the strategy iteration mechanism of reinforcement learning to update the strategy to adapt to data changes and meet higher data processing standards:

[0073]

[0074] Among them, θ t is the policy parameter at time t, α is the learning rate, and δ t is the time difference error, which represents the difference between the actual reward and the expected reward, is the policy gradient.

[0075] Optionally, the S5 specifically includes:

[0076] S51. Use the variational autoencoder to process the input data, extract data features and generate feature representations that can be used for model optimization. The encoder part converts the input data x into a latent space representation z:

[0077]

[0078] Where μ and σ are the encoded mean and standard deviation, ∈ is the random noise sampled from the standard normal distribution, Represents element-wise multiplication;

[0079] S52. Automatically adjust the network structure and hyperparameters of the model through genetic algorithms, and initialize the population to represent different network structures and parameter configurations:

[0080]

[0081] Where x represents the model under a certain network configuration, and error(x) represents the error rate of the model under this configuration on the validation data set;

[0082] S53. The genetic algorithm iterates to select and optimize the model. In each generation, the genetic algorithm uses selection, crossover, and mutation operations to generate new individuals and form a new population. The roulette wheel selection method is used to select two parent individuals from the current population according to the fitness function:

[0083]

[0084] Among them, p i is the probability, f(x i ) is the fitness of individual i, n is the population size;

[0085] Apply single-point crossover to the selected parent individuals, the crossover point k is randomly selected, and the characteristics of the offspring are inherited from the two parents:

[0086]

[0087] Among them, x1 and x2 are parent individuals, y1 and y2 are generated offspring individuals, and i represents the index of individual characteristics;

[0088] Randomly modify certain characteristics of offspring individuals with a certain mutation probability μ:

[0089] y′ i =y i +δ,ifrandom<μ;

[0090] Among them, y i is the unmutated feature value, δ is the mutation amount, usually a random value within a small range, and random represents the generated random number used to decide whether to mutate;

[0091] S54. Continuously monitor the progress of the genetic algorithm and evaluate the performance of each generation of models. The final selected model structure and parameter configuration will be used for the actual model training task. The application value of the model will be greatly improved through intelligent optimization methods. The final model parameters θ obtained by genetic algorithm optimization * :

[0092]

[0093] Where F1(θ) represents the F1 score calculated on the validation set using the parameter set θ;

[0094] Using the selected parameter set θ *Train the final model and use the cross entropy loss function to optimize the model:

[0095]

[0096] Where L is the loss function, N is the number of training data, and y i is the true label of the i-th sample, is the model parameter θ * The predicted output for the i-th sample.

[0097] Optionally, the S6 specifically includes:

[0098] S61. Deploy a dynamic task scheduling system based on priority and resource utilization efficiency. The system initially collects the demand information of each task, including the urgency of the task, resource requirements and expected execution time. Each task is assigned a priority score:

[0099] P = α × U + β × R;

[0100] Among them, P represents the priority score of the task, U represents the urgency of the task, R represents the resource requirement, and α and β are weight factors;

[0101] S62. The system monitors the usage of computing resources in real time. According to the real-time usage of resources and the priority of tasks, the scheduling system dynamically adjusts the task queue:

[0102]

[0103] Among them, R alloc is the amount of resources allocated to task i, R avail is the total amount of resources currently available, P i is the priority score of task i, Q is the current task queue;

[0104] S63. When existing tasks are completed, the scheduling system will re-evaluate and update the task queue to reflect the latest task priority and resource status. The system implements a feedback control mechanism to dynamically adjust the priority allocation and resource allocation strategy based on the efficiency and completion of task execution;

[0105] S64. The system efficiently executes all tasks according to priority, while maximizing resource utilization and reducing task waiting time. Through this dynamic scheduling strategy, the system can flexibly respond to real-time changes in computing resources and processing requirements, and optimize the overall process of data processing and model training.

[0106] The beneficial effects of the present invention are:

[0107] (1) The present invention integrates deep learning, natural language processing technology, genetic algorithms and dynamic task scheduling systems to provide automatic recognition and intelligent processing of data structures and types, enabling the system to automatically build database tables and optimize data models to adapt to changing data requirements and processing tasks. This process greatly reduces the workload of manual table building and model selection, and improves the efficiency and accuracy of data processing.

[0108] (2) The present invention improves data quality and reduces errors and deviations in subsequent analysis through automatic data preprocessing and cleaning steps. The automated cleaning process dynamically adjusts the preprocessing strategy according to the type and quality of the data to ensure that the data reaches the optimal state before entering the analysis phase, which is crucial to ensure the reliability and effectiveness of the data analysis results.

[0109] (3) The present invention optimizes the allocation and utilization of computing resources through a dynamic task scheduling system, ensuring that high-priority tasks can obtain the necessary resources for processing in a timely manner. The real-time monitoring and automatic adjustment functions of this system enable data processing and model training tasks to run efficiently even in a resource-limited environment, thereby improving the response speed and processing capacity of the overall system.

[0110] (4) The present invention combines a variety of advanced technologies to achieve full process automation from data input to model deployment, significantly improving the intelligence level of data processing tasks. This not only provides users with a more convenient and efficient data processing solution, but also provides powerful technical support for complex problems in large-scale data environments, enabling enterprises and organizations to better utilize their data resources and promote information development. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0112] Figure 1 This is an overall flow chart of the one-click table building and model processing task generation method based on deep learning proposed by the present invention;

[0113] Figure 2 A detailed flow chart of the data structure and type automatic identification of the one-click table building and model processing task generation method based on deep learning proposed by the present invention;

[0114] Figure 3 This is a flow chart of the genetic algorithm for optimizing model performance of the one-click table building and model processing task generation method based on deep learning proposed by the present invention. DETAILED DESCRIPTION

[0115] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0116] refer to Figure 1-Figure 3 , a one-click table building and model processing task generation method based on deep learning, characterized in that it includes the following steps:

[0117] S1. Use deep learning algorithms to automatically analyze structured and unstructured input data and extract the structure and type of data from the user's natural language instructions;

[0118] In this implementation, S1 specifically includes:

[0119] S11. The system first receives input including structured and unstructured data. The system receives instructions provided by the user in natural language, indicating the requirements for data processing and table creation;

[0120] S12. For unstructured image data, use convolutional neural network to extract features:

[0121]

[0122] Among them, F(x,y) represents the feature output of the image at position (x,y), K represents the convolution kernel, I represents the input image data, i and j are the index positions in the convolution kernel, and k is the size of the convolution kernel;

[0123] S13. For text data, the system uses long short-term memory network technology for semantic analysis and structural parsing:

[0124] h t =σ(W ih ωx t +b ih +W hh ·h (t-1) +b hh );

[0125] Among them, h t is the hidden state at time step t, x t is the input vector at time step t, W ih and W hh are the weight matrices from input to hidden and hidden to hidden, respectively, and b ih and b hh is the bias term, σ is the activation function;

[0126] S14. Use the support vector machine decision model to combine the features extracted from the image through the convolutional neural network and the features processed from the text through the long short-term memory network to determine the data structure and type:

[0127] f(x)=sgn(w T φ(x)+b);

[0128] Among them, x represents the feature vector, which is composed of the features of image and text data, φ(x) is the function that maps the feature vector to a high-dimensional space, w is the weight vector, b is the bias term, and sgn is the sign function used to determine which category the data belongs to;

[0129] S15. Based on the data structure and type determined by the support vector machine model, the system automatically maps to the corresponding database model, selects a suitable database table structure for each data type, and configures specific data field types. The system automatically generates SQL statements and non-SQL database commands based on the output of the integrated model:

[0130] CREATETABLEIFNOTEXISTStable_name(column_namedata_type,…).

[0131] S2. The data relationship intelligent recognition system enhanced by the graph neural network automatically builds and optimizes the database table structure based on the analysis and extraction results, and performs intelligent matching and configuration of data fields and data relationships;

[0132] In this implementation, S2 specifically includes:

[0133] S21. Use graph neural networks to process input data and identify complex relationships and dependencies between data elements:

[0134]

[0135] in, represents the feature vector of node v at time step t, N(v) represents the set of neighbor nodes of node v, c vu is the contribution weight of node u to node v, W (t) is the weight matrix at time step t, is the input feature of node v at time step t, and f is the activation function;

[0136] S22. Automatically identify key entities and their relationships in the data by analyzing the output of the graph neural network:

[0137] r vu =softmax(W r ·concat(h v ,h u ));

[0138] Among them, r vu represents the relationship type from entity v to entity u, Wr is the weight matrix for relationship classification, concat(h v ,h u ) is the result of concatenating the feature vectors of entity v and entity u. The softmax function is used to normalize the output into a probability distribution, indicating the possibility of different types of relationships.

[0139] S23. Automatically construct the database table structure based on the data relationships and entity features obtained from the graph neural network, and apply optimization algorithms to select the best table structure and data field type:

[0140]

[0141] Among them, S represents the selected storage structure, T represents the table configuration, V is the set of data entities, E is the set of relationships between entities, and S v is the storage structure of entity v, D v is the data type of entity v, L is the loss function for the mismatch between storage structure and data type, T vu is the table configuration between entities v and u, R vu is the type of relationship between entities, C is the cost function for mismatch between configuration and relationship type, λ1 and λ2 are weight parameters for adjusting the impact of the two losses;

[0142] S24. The system automatically optimizes the index and storage configuration of the database table based on the output of the graph neural network model to improve query efficiency and data processing speed, evaluate the performance of different index strategies, and select the strategy that best suits the current data structure:

[0143] m I (α·Cost storage (I)+β·Cost query (I));

[0144] Among them, I represents the index configuration set, Cost storage (I) is the storage cost under a given index configuration, Cost query (I) is the query cost under the index configuration, α and β are weight parameters to balance the impact of storage and query costs;

[0145] S25. Automatically generate SQL statements and corresponding commands applicable to non-SQL databases, create and configure database tables, and convert optimization results into specific database commands:

[0146]

[0147] Among them, table_name is the table name determined according to the output of the graph neural network, n is the number of fields, column_name iis the name of the i-th field, data_type i Is the data type of the i-th field.

[0148] S3, using an integrated multimodal data fusion framework to process and integrate information from different data sources;

[0149] In this implementation, S3 specifically includes:

[0150] S31. Use an integrated multimodal data fusion framework to process and integrate information from different data sources, including text, image, sound, and video data:

[0151] F integrated =concat(F text ,F image ,F audio ,F video );

[0152] Among them, F integrated represents the fused feature vector, F text , F image , F audio , and F video They represent feature vectors extracted from text, image, audio and video data respectively, and concat represents the concatenation function of feature vectors;

[0153] S32. Analyze the fused feature vectors, identify and extract key instructions for data processing and table building, and use variational autoencoders to further extract features and reduce dimensions:

[0154]

[0155] Where z represents the potential feature space, μ and σ are the encoded mean and variance, ∈ is the noise sampled from the standard normal distribution, Represents element-wise multiplication;

[0156] S33, using the extracted feature vectors to automatically generate task instructions for data processing and table building, using a multi-layer perceptron for feature analysis and task generation:

[0157] y=softmax(W2·σ(W1·x+b1)+b2);

[0158] Among them, x represents the comprehensive feature vector of the input, W1 and W2 are the weight matrices of the network layer, b1 and b2 are bias vectors, σ represents the activation function, and the softmax function is used to convert the network output into a probability distribution, which represents the possibility of table creation and data processing tasks in different databases.

[0159] S4. Introduce an automatic data cleaning and preprocessing system based on deep reinforcement learning to automatically select and apply the optimal cleaning and preprocessing strategy based on data type and quality;

[0160] In this implementation, S4 specifically includes:

[0161] S41. Use an algorithm based on deep reinforcement learning to automatically analyze the type and quality of input data, use state representation learning to encode the data set, and convert the state s of the data into a low-dimensional feature representation through an encoding function φ:

[0162] s ′ =φ(s);

[0163] Among them, s represents the original data state, s ′ Indicates the encoded state;

[0164] S42. According to the encoded data state, use the deep reinforcement learning model to select the most suitable data cleaning and preprocessing strategy, and maximize the long-term reward by optimizing the decision strategy π:

[0165]

[0166] Among them, π * represents the optimal strategy, R t is the reward obtained at time t, γ is the discount factor, which represents the current value of future rewards;

[0167] S43. Apply the selected strategy to perform data cleaning and preprocessing tasks, using statistical analysis-based methods to automatically identify and remove outliers in the data:

[0168]

[0169] Where X is the observed value, μ is the mean, and σ is the standard deviation;

[0170] Choose an appropriate filling strategy based on the type and distribution of the data. For numerical data, use mean filling:

[0171] X filled =X missing ∪μ;

[0172] Among them, X missing is the missing data point, μ is the mean of the non-missing data points, and ∪ represents the replacement operation;

[0173] Standardize the data to eliminate dimension effects and improve algorithm performance:

[0174]

[0175] Where X is the original data, min(X) and max(X) are the minimum and maximum values ​​of the data respectively;

[0176] S44. Continuously monitor the effect of data preprocessing and adjust the strategy according to the actual effect. Use the strategy iteration mechanism of reinforcement learning to update the strategy to adapt to data changes and meet higher data processing standards:

[0177]

[0178] Among them, θ t is the policy parameter at time t, α is the learning rate, and δ t is the time difference error, which represents the difference between the actual reward and the expected reward, is the policy gradient.

[0179] S5. Optimize model performance through genetic algorithms, use variational autoencoders to process data characteristics, and automatically adjust the model's network structure and hyperparameters;

[0180] In this implementation, S5 specifically includes:

[0181] S51. Use the variational autoencoder to process the input data, extract data features and generate feature representations that can be used for model optimization. The encoder part converts the input data x into a latent space representation z:

[0182]

[0183] Where μ and σ are the encoded mean and standard deviation, ∈ is the random noise sampled from the standard normal distribution, Represents element-wise multiplication;

[0184] S52. Automatically adjust the network structure and hyperparameters of the model through genetic algorithms, and initialize the population to represent different network structures and parameter configurations:

[0185]

[0186] Where x represents the model under a certain network configuration, and error(x) represents the error rate of the model under this configuration on the validation data set;

[0187] S53. The genetic algorithm iterates to select and optimize the model. In each generation, the genetic algorithm uses selection, crossover, and mutation operations to generate new individuals and form a new population. The roulette wheel selection method is used to select two parent individuals from the current population according to the fitness function:

[0188]

[0189] Among them, p i is the probability, f(xi ) is the fitness of individual i, n is the population size;

[0190] Apply single-point crossover to the selected parent individuals, the crossover point k is randomly selected, and the characteristics of the offspring are inherited from the two parents:

[0191]

[0192] Among them, x1 and x2 are parent individuals, y1 and y2 are generated offspring individuals, and i represents the index of individual characteristics;

[0193] Randomly modify certain characteristics of offspring individuals with a certain mutation probability μ:

[0194] y′ i =y i +δ,ifrandom<μ;

[0195] Among them, y i is the unmutated feature value, δ is the mutation amount, usually a random value within a small range, and random represents the generated random number used to decide whether to mutate;

[0196] S54. Continuously monitor the progress of the genetic algorithm and evaluate the performance of each generation of models. The final selected model structure and parameter configuration will be used for the actual model training task. The application value of the model will be greatly improved through intelligent optimization methods. The final model parameters θ obtained by genetic algorithm optimization * :

[0197]

[0198] Where F1(θ) represents the F1 score calculated on the validation set using the parameter set θ;

[0199] Using the selected parameter set θ * Train the final model and use the cross entropy loss function to optimize the model:

[0200]

[0201] Where L is the loss function, N is the number of training data, and y i is the true label of the i-th sample, is the model parameter θ * The predicted output for the i-th sample.

[0202] S6. Based on task priority and resource utilization efficiency, implement dynamic scheduling algorithms to optimize computing resource allocation in real time, reduce processing delays, and improve system response speed and task processing capabilities.

[0203] In this implementation, S6 specifically includes:

[0204] S61. Deploy a dynamic task scheduling system based on priority and resource utilization efficiency. The system initially collects the demand information of each task, including the urgency of the task, resource requirements and expected execution time. Each task is assigned a priority score:

[0205] P = α × U + β × R;

[0206] Among them, P represents the priority score of the task, U represents the urgency of the task, R represents the resource requirement, and α and β are weight factors;

[0207] S62. The system monitors the usage of computing resources in real time. According to the real-time usage of resources and the priority of tasks, the scheduling system dynamically adjusts the task queue:

[0208]

[0209] Among them, R alloc is the amount of resources allocated to task i, R avail is the total amount of resources currently available, P i is the priority score of task i, Q is the current task queue;

[0210] S63. When existing tasks are completed, the scheduling system will re-evaluate and update the task queue to reflect the latest task priority and resource status. The system implements a feedback control mechanism to dynamically adjust the priority allocation and resource allocation strategy based on the efficiency and completion of task execution;

[0211] S64. The system efficiently executes all tasks according to priority, while maximizing resource utilization and reducing task waiting time. Through this dynamic scheduling strategy, the system can flexibly respond to real-time changes in computing resources and processing requirements, and optimize the overall process of data processing and model training.

[0212] Embodiment 1:

[0213] In order to verify the beneficial effects of the present invention, we chose to implement an example in a large retail chain enterprise. The enterprise has huge data processing needs, including daily sales, inventory management, customer behavior analysis and other information processing. Before implementation, the enterprise faced problems of low data processing efficiency and delayed decision-making, especially during peak sales periods, when data accumulation made real-time analysis almost impossible to perform.

[0214] By deploying the one-click table creation and model processing task generation method based on deep learning proposed in this invention, the enterprise has achieved significant optimization of data processing and analysis. The following is the specific data performance after the implementation of the invention. The statistical results show that the enterprise's data processing capabilities have been significantly improved before and after the deployment of this system.

[0215] Table 1 Retail enterprise data processing capability improvement report

[0216]

[0217]

[0218] As can be seen from Table 1, after the system was deployed, the data processing time was shortened from an average of 8 hours to 2 hours, and the automation rate of data preprocessing increased from 20% to 90%, significantly improving the efficiency of preliminary data preparation. The model deployment time was also significantly shortened from 12 hours to 3 hours, which enabled data analysis to be transformed into viable business strategies more quickly. In addition, the improvement in system response speed and data analysis accuracy effectively supported rapid decision-making, and the improvement in computing resource utilization also demonstrated the cost-effectiveness brought by system optimization.

[0219] Most importantly, the improvement of real-time data analysis capabilities enables enterprises to process larger data sets, increasing the real-time processing capacity from 1TB to 5TB, greatly enhancing the competitiveness of enterprises in the data-driven market environment. This improvement not only improves the operational efficiency of enterprises, but also enhances market adaptability and customer satisfaction, and improves overall business benefits and strategic decision-making capabilities of enterprises through efficient data processing.

[0220] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A one-click table building and model processing task generation method based on deep learning, characterized in that: The steps include: S1. Use deep learning algorithms to automatically analyze structured and unstructured input data and extract the structure and type of data from the user's natural language instructions; S2. The data relationship intelligent recognition system enhanced by the graph neural network automatically builds and optimizes the database table structure based on the analysis and extraction results, and performs intelligent matching and configuration of data fields and data relationships; S3, using an integrated multimodal data fusion framework to process and integrate information from different data sources; S4. Introduce an automatic data cleaning and preprocessing system based on deep reinforcement learning to automatically select and apply the optimal cleaning and preprocessing strategy based on data type and quality; S5. Optimize model performance through genetic algorithms, use variational autoencoders to process data characteristics, and automatically adjust the model's network structure and hyperparameters; S6. Based on task priority and resource utilization efficiency, implement dynamic scheduling algorithms to optimize computing resource allocation in real time, reduce processing delays, and improve system response speed and task processing capabilities.

2. The one-click table building and model processing task generation method based on deep learning according to claim 1 is characterized in that: The S1 specifically includes: S11. The system first receives input including structured and unstructured data. The system receives instructions provided by the user in natural language, indicating the requirements for data processing and table creation; S12. For unstructured image data, use convolutional neural network to extract features: Among them, F(x,y) represents the feature output of the image at position (x,y), K represents the convolution kernel, I represents the input image data, i and j are the index positions in the convolution kernel, and k is the size of the convolution kernel; S13. For text data, the system uses long short-term memory network technology for semantic analysis and structural parsing: h t =σ(W ih ·x t +b ih +W hh ·h (t-1) +b hh ); Among them, h t is the hidden state at time step t, x t is the input vector at time step t, W ih and W hh are the weight matrices from input to hidden and hidden to hidden, respectively, and b ih and b hh is the bias term, σ is the activation function; S14. Use the support vector machine decision model to combine the features extracted from the image through the convolutional neural network and the features processed from the text through the long short-term memory network to determine the data structure and type: f(x)=sgn(w T φ(x)+b); Among them, x represents the feature vector, which is composed of the features of image and text data, φ(x) is the function that maps the feature vector to a high-dimensional space, w is the weight vector, b is the bias term, and sgn is the sign function used to determine which category the data belongs to; S15. Based on the data structure and type determined by the support vector machine model, the system automatically maps to the corresponding database model, selects a suitable database table structure for each data type, and configures specific data field types. The system automatically generates SQL statements and non-SQL database commands based on the output of the integrated model: CREATETABLEIFNOTEXISTStable_name(column_namedata_type,…).

3. The one-click table building and model processing task generation method based on deep learning according to claim 2 is characterized in that: The S2 specifically includes: S21. Use graph neural networks to process input data and identify complex relationships and dependencies between data elements: in, represents the feature vector of node v at time step t, N(v) represents the set of neighbor nodes of node v, c vu is the contribution weight of node u to node v, W (t) is the weight matrix at time step t, is the input feature of node v at time step t, and f is the activation function; S22. Automatically identify key entities and their relationships in the data by analyzing the output of the graph neural network: r vu =softmax(W r ·concat(h v ,h u )); Among them, r vu represents the relationship type from entity v to entity u, W r is the weight matrix for relationship classification, concat(h v ,h u ) is the result of concatenating the feature vectors of entity v and entity u. The softmax function is used to normalize the output into a probability distribution, indicating the possibility of different types of relationships. S23. Automatically construct the database table structure based on the data relationships and entity features obtained from the graph neural network, and apply optimization algorithms to select the best table structure and data field type: Among them, S represents the selected storage structure, T represents the table configuration, V is the set of data entities, E is the set of relationships between entities, and S v is the storage structure of entity v, D v is the data type of entity v, L is the loss function for the mismatch between storage structure and data type, T vu is the table configuration between entities v and u, R vu is the type of relationship between entities, C is the cost function for mismatch between configuration and relationship type, λ1 and λ2 are weight parameters for adjusting the impact of the two losses; S24. The system automatically optimizes the index and storage configuration of the database table based on the output of the graph neural network model to improve query efficiency and data processing speed, evaluate the performance of different index strategies, and select the strategy that best suits the current data structure: m I (α·Cost storage (I)+β·Cost query (I)); Among them, I represents the index configuration set, Cost storage (I) is the storage cost under a given index configuration, Cost query (I) is the query cost under the index configuration, α and β are weight parameters to balance the impact of storage and query costs; S25. Automatically generate SQL statements and corresponding commands applicable to non-SQL databases, create and configure database tables, and convert optimization results into specific database commands: Among them, table_name is the table name determined according to the output of the graph neural network, n is the number of fields, column_name i is the name of the i-th field, data_type i Is the data type of the i-th field.

4. The one-click table building and model processing task generation method based on deep learning according to claim 3 is characterized in that: The S3 specifically includes: S31. Use an integrated multimodal data fusion framework to process and integrate information from different data sources, including text, image, sound, and video data: F integrated =concat(F text ,F image ,F audio ,F video ); Among them, F integrated represents the fused feature vector, F text , F image , F audio , and F video They represent feature vectors extracted from text, image, audio and video data respectively, and concat represents the concatenation function of feature vectors; S32. Analyze the fused feature vectors, identify and extract key instructions for data processing and table building, and use variational autoencoders to further extract features and reduce dimensions: Where z represents the potential feature space, μ and σ are the encoded mean and variance, ∈ is the noise sampled from the standard normal distribution, Represents element-wise multiplication; S33, using the extracted feature vectors to automatically generate task instructions for data processing and table building, using a multi-layer perceptron for feature analysis and task generation: y=softmax(W2·σ(W1·x+b1)+b2); Among them, x represents the comprehensive feature vector of the input, W1 and W2 are the weight matrices of the network layer, b1 and b2 are bias vectors, σ represents the activation function, and the softmax function is used to convert the network output into a probability distribution, which represents the possibility of table creation and data processing tasks in different databases.

5. The one-click table building and model processing task generation method based on deep learning according to claim 4 is characterized in that: The S4 specifically includes: S41. Use an algorithm based on deep reinforcement learning to automatically analyze the type and quality of input data, use state representation learning to encode the data set, and convert the state s of the data into a low-dimensional feature representation through an encoding function φ: s ′ =φ(s); Among them, s represents the original data state, s ′ Indicates the encoded state; S42. According to the encoded data state, use the deep reinforcement learning model to select the most suitable data cleaning and preprocessing strategy, and maximize the long-term reward by optimizing the decision strategy π: Among them, π * represents the optimal strategy, R t is the reward obtained at time t, γ is the discount factor, which represents the current value of future rewards; S43. Apply the selected strategy to perform data cleaning and preprocessing tasks, using statistical analysis-based methods to automatically identify and remove outliers in the data: Where X is the observed value, μ is the mean, and σ is the standard deviation; Choose an appropriate filling strategy based on the type and distribution of the data. For numerical data, use mean filling: X filled =X missing ∪μ; Among them, X missing is the missing data point, μ is the mean of the non-missing data points, and ∪ represents the replacement operation; Standardize the data to eliminate dimension effects and improve algorithm performance: Where X is the original data, min(X) and max(X) are the minimum and maximum values ​​of the data respectively; S44. Continuously monitor the effect of data preprocessing and adjust the strategy according to the actual effect. Use the strategy iteration mechanism of reinforcement learning to update the strategy to adapt to data changes and meet higher data processing standards: Among them, θ t is the policy parameter at time t, α is the learning rate, and δ t is the time difference error, which represents the difference between the actual reward and the expected reward, is the policy gradient.

6. The one-click table building and model processing task generation method based on deep learning according to claim 5 is characterized in that: The S5 specifically includes: S51. Use the variational autoencoder to process the input data, extract data features and generate feature representations that can be used for model optimization. The encoder part converts the input data x into a latent space representation z: Where μ and σ are the encoded mean and standard deviation, ∈ is the random noise sampled from the standard normal distribution, Represents element-wise multiplication; S52. Automatically adjust the network structure and hyperparameters of the model through genetic algorithms, and initialize the population to represent different network structures and parameter configurations: Where x represents the model under a certain network configuration, and error(x) represents the error rate of the model under this configuration on the validation data set; S53. The genetic algorithm iterates to select and optimize the model. In each generation, the genetic algorithm uses selection, crossover, and mutation operations to generate new individuals and form a new population. The roulette wheel selection method is used to select two parent individuals from the current population according to the fitness function: Among them, p i is the probability, f(x i ) is the fitness of individual i, n is the population size; Apply single-point crossover to the selected parent individuals, the crossover point k is randomly selected, and the characteristics of the offspring are inherited from the two parents: Among them, x1 and x2 are parent individuals, y1 and y2 are generated offspring individuals, and i represents the index of individual characteristics; Randomly modify certain characteristics of offspring individuals with a certain mutation probability μ: and' i =and i +δ,ifrandom<μ; Among them, y i is the unmutated feature value, δ is the mutation amount, usually a random value within a small range, and random represents the generated random number used to decide whether to mutate; S54. Continuously monitor the progress of the genetic algorithm and evaluate the performance of each generation of models. The final selected model structure and parameter configuration will be used for the actual model training task. The application value of the model will be greatly improved through intelligent optimization methods. The final model parameters θ obtained by genetic algorithm optimization * : Where F1(θ) represents the F1 score calculated on the validation set using the parameter set θ; Using the selected parameter set θ * Train the final model and use the cross entropy loss function to optimize the model: Where L is the loss function, N is the number of training data, and y i is the true label of the i-th sample, is the model parameter θ * The predicted output for the i-th sample.

7. The one-click table building and model processing task generation method based on deep learning according to claim 6 is characterized in that: The S6 specifically includes: S61. Deploy a dynamic task scheduling system based on priority and resource utilization efficiency. The system initially collects the demand information of each task, including the urgency of the task, resource requirements and expected execution time. Each task is assigned a priority score: P = α × U + β × R; Among them, P represents the priority score of the task, U represents the urgency of the task, R represents the resource requirement, and α and β are weight factors; S62. The system monitors the usage of computing resources in real time. According to the real-time usage of resources and the priority of tasks, the scheduling system dynamically adjusts the task queue: Among them, R alloc is the amount of resources allocated to task i, R avail is the total amount of resources currently available, P i is the priority score of task i, Q is the current task queue; S63. When existing tasks are completed, the scheduling system will re-evaluate and update the task queue to reflect the latest task priority and resource status. The system implements a feedback control mechanism to dynamically adjust the priority allocation and resource allocation strategy based on the efficiency and completion of task execution; S64. The system efficiently executes all tasks according to priority, while maximizing resource utilization and reducing task waiting time. Through this dynamic scheduling strategy, the system can flexibly respond to real-time changes in computing resources and processing requirements, and optimize the overall process of data processing and model training.

Images