Mixed data platform based on artificial intelligence

Abstract

The invention provides a hybrid data platform based on artificial intelligence, and the platform comprises a hybrid data platform which comprises an ontology hierarchy module, a logic hierarchy module, an analysis hierarchy module, an operation hierarchy module, an interpretation hierarchy module, and a language hierarchy module. The ontology hierarchy module is used for original data, domain knowledge, data processing, algorithm and data modeling; the logic hierarchy module is used for data management; the analysis hierarchy module is used for hierarchy integration; the operation level module is used for data collection, data preprocessing, data transmission, data aggregation, data storage, data analysis and data interpretation. According to the artificial intelligence-based mixed data platform provided by the invention, in the aspect of original data, the source reliability of the data is ensured, the most basic material is provided for subsequent data processing and analysis, various types of original data can be collected from different channels, the diversity of the data is enriched, and the data processing efficiency is improved. And the possibility is provided for more comprehensive analysis and modeling.

Description

Technical Field

[0001] The present invention relates to the field of data platforms, and in particular to a hybrid data platform based on artificial intelligence. Background Art

[0002] With the rapid development of information technology, the scale and complexity of data are increasing. Traditional data processing platforms can no longer meet the growing demand. The background technology mainly includes the following aspects:

[0003] The rapid growth of data volume

[0004] In today's digital age, large amounts of data are constantly being generated in all fields. Internet companies have to process massive amounts of user behavior data, transaction data, and log data every day. Scientific research institutions also collect a large amount of experimental data and observation data when conducting scientific research. According to statistics, the total amount of global data doubles every two years, and the growth rate of data far exceeds the rate of improvement in storage and processing capabilities.

[0005] Diversity of data types

[0006] Not only is the amount of data huge, but the types are also becoming increasingly diverse. In addition to traditional structured data, it also includes a large amount of semi-structured data and unstructured data.

[0007] Different types of data require different processing methods and technologies, and traditional data platforms often find it difficult to effectively integrate and process these diverse data.

[0008] Limitations of Traditional Data Platforms

[0009] Limited data storage and processing capabilities: Traditional data platforms usually use relational databases or data warehouses to store data. These technologies have performance bottlenecks when processing large-scale data. Relational databases have poor scalability and are difficult to cope with the rapid growth of data volume; although data warehouses can store and analyze large-scale data, they have limited processing capabilities for unstructured data.

[0010] In addition, traditional data platforms also have difficulties in processing real-time data and streaming data, and it is difficult to meet application scenarios with high requirements for data real-time performance.

[0011] Difficulty in data integration and analysis: Traditional data platforms can only process a single type of data and find it difficult to effectively integrate and analyze diverse data from different data sources. At the same time, traditional data platforms usually require professional data analysts to perform complex SQL queries and programming when conducting data analysis, which is difficult for ordinary users.

[0012] Lack of intelligent analysis capabilities: Traditional data platforms mainly provide data storage and basic data analysis functions, and lack intelligent analysis capabilities.

[0013] Therefore, it is necessary to provide an artificial intelligence-based hybrid data platform to solve the above technical problems. Summary of the invention

[0014] The present invention provides a hybrid data platform based on artificial intelligence, which solves the problems of limited data storage and processing capabilities and difficulty in data integration and analysis of traditional hybrid data platforms.

[0015] In order to solve the above technical problems, the present invention provides a hybrid data platform based on artificial intelligence, comprising:

[0016] A hybrid data platform, the hybrid data platform comprising an ontology level module, a logic level module, an analysis level module, an operation level module, an interpretation level module and a language level module;

[0017] The ontology level modules are used for raw data, domain knowledge, data processing, algorithms and data modeling;

[0018] The logic level module is used for data management;

[0019] The analysis level module is used for level integration;

[0020] The operational level modules are used for data collection, data preprocessing, data transmission, data aggregation, data storage, data analysis and data interpretation.

[0021] Preferably, the language level modules are used to construct a case-based grammar for the attributes of various ontological concepts.

[0022] Preferably, the explanation level module is used to generate reasonable explanations based on the causal dependencies and their logical representations at the ontological, logical and analytical levels, and the data processing results at the operational level.

[0023] Preferably, the logic level module references concepts and individuals of the ontology level module by specifying heuristic rules that control the execution of data management operations.

[0024] Preferably, the data preprocessing includes data cleaning, data conversion, data dimensionality reduction and data integration, the data cleaning includes processing missing values ​​and processing outliers, the data conversion includes standardization, normalization and logarithmic transformation, the data dimensionality reduction includes principal component analysis and singular value decomposition, and the data integration includes entity recognition and data conflict resolution.

[0025] Preferably, the method of processing missing values ​​includes deletion and filling, and the filling method includes mean filling, median filling and most frequent value filling.

[0026] Preferably, the processing of outliers includes a capping method and a box plot method, and the box plot method utilizes a box plot to identify outliers.

[0027] Preferably, the data modeling includes model selection, data partitioning, hyperparameter adjustment, model integration and regularization, the model selection includes considering the problem type and evaluating model performance, the data partitioning includes training, validation and test sets and cross-validation, and the hyperparameter adjustment includes grid search, random search and Bayesian optimization.

[0028] Preferably, the types of problems considered include classification problems, regression problems and clustering problems.

[0029] Preferably, the regression problem is used for the task of predicting continuous values.

[0030] Compared with related technologies, the hybrid data platform based on artificial intelligence provided by the present invention has the following beneficial effects:

[0031] The present invention provides a hybrid data platform based on artificial intelligence.

[0032] Raw data: It ensures the reliability of the data source and provides the most basic material for subsequent data processing and analysis. It can collect various types of raw data from different channels, enrich the diversity of data, and provide the possibility for more comprehensive analysis and modeling.

[0033] Domain knowledge: Introducing domain knowledge can provide professional guidance for data processing and modeling. For example, in the medical field, doctors’ expertise can help the platform better understand the meaning and value of medical data and improve the accuracy of disease diagnosis and treatment plan recommendations.

[0034] Data processing: The direction and methods of data processing are clarified. By integrating various data processing technologies, such as cleaning, conversion, and normalization, the quality and availability of data can be improved.

[0035] Algorithms and data modeling: It provides a rich selection of algorithms and modeling methods, and can select the most appropriate algorithms and models according to different problems and data characteristics, thus improving the accuracy and generalization ability of the models;

[0036] Data management: It realizes efficient organization and management of data, including data storage, retrieval, and update, which can ensure the security and integrity of data and prevent data loss and damage;

[0037] Hierarchical integration: effectively integrates data and information at different levels, achieving seamless connection from raw data to final analysis results, avoiding data loss and errors during transmission and conversion between different levels, and improving data consistency and accuracy;

[0038] Data collection: It can collect data from a variety of data sources, including sensors, databases, file systems, networks, etc., enriching the sources and types of data. It can collect data in real time or regularly to ensure the timeliness and freshness of the data.

[0039] Data preprocessing: Through data cleaning, conversion, normalization and other processing, noise and outliers in the data can be removed, the quality and availability of the data can be improved, and data in different formats can be converted into a unified format to facilitate subsequent analysis and processing;

[0040] Data transmission: ensure fast and secure data transmission between different modules and systems, and use encryption and data compression technologies to improve the efficiency and security of data transmission;

[0041] Data aggregation: Integrate and summarize scattered data to form more valuable information. Through data fusion, association analysis and other technologies, the potential relationships and patterns between data can be discovered;

[0042] Data storage: It provides reliable data storage solutions, including distributed storage, cloud storage, etc., which can ensure the security and persistence of data and prevent data loss and damage;

[0043] Data analysis: Using various data analysis techniques, such as statistical analysis, machine learning, and deep learning, to extract valuable information and knowledge from data, we can discover potential patterns and trends in the data and provide support for decision-making;

[0044] Data interpretation: Explain and illustrate the data analysis results so that users can understand the meaning and value of the data, which can help users better use the data analysis results to make decisions and improve the scientificity and rationality of decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the structure of a first embodiment of a hybrid data platform based on artificial intelligence provided by the present invention;

[0046] Figure 2 A schematic diagram of the structure of a second embodiment of a hybrid data platform based on artificial intelligence provided by the present invention;

[0047] Figure 3 A schematic structural diagram of a third embodiment of an artificial intelligence-based hybrid data platform provided by the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0049] First embodiment

[0050] Please refer to Figure 1 ,in, Figure 1 A structural diagram of a first embodiment of a hybrid data platform based on artificial intelligence provided by the present invention. A hybrid data platform based on artificial intelligence includes:

[0051] A hybrid data platform, the hybrid data platform comprising an ontology level module, a logic level module, an analysis level module, an operation level module, an interpretation level module and a language level module;

[0052] The ontology level modules are used for raw data, domain knowledge, data processing, algorithms and data modeling;

[0053] The logic level module is used for data management;

[0054] The analysis level module is used for level integration;

[0055] The operational level modules are used for data collection, data preprocessing, data transmission, data aggregation, data storage, data analysis and data interpretation.

[0056] The basic concepts of the ontology level modules include situations, which describe the static state of things; events, which formalize the asynchronous results of data processing; and actions, which model synchronous data operations. At this level, the model is expressed in the form of OWL ontology.

[0057] Logical level modules are modeled using SWRL rules.

[0058] The analysis level module integrates the first two levels into the operational workflow and models it in the form of an RDF graph. This workflow is much simpler than the BPEL workflow and does not require the use of web service APIs for remote orchestration of services.

[0059] The operation level module enables each operation to be implemented as a software component, configured and deployed into cloud containers using metadata in the ontology, and controlled through workflow monitoring tools available on the platform.

[0060] The language level modules form a case-based grammar for the attributes of various ontological concepts.

[0061] The explanation level module is used to generate reasonable explanations based on the causal dependencies and their logical representations at the ontology, logic and analysis levels, as well as the data processing results at the operation level.

[0062] Explanation Level Module It is based on a single ontology that can be extended to different domains and problems and contains domain- and problem-specific concepts, roles, and heuristic rules, especially for explanatory purposes.

[0063] The logical level module references concepts and individuals of the ontology level module by specifying heuristic rules that control the execution of data management operations.

[0064] Language-Level Modules The top-level concepts of the ontology hierarchy are the OWL classes Problem, Data, and Solution, which represent domain-specific and problem-specific information.

[0065] The classification of infrastructure and resources class describes the software and hardware components and tools available on the platform for data processing.

[0066] At the logical level, the model can be further extended with domain-specific, problem-specific, and application-specific heuristic rules.

[0067] From the OWL and SWRL representations, we can extract information to build a pure RDF graph.

[0068] Compared with related technologies, the hybrid data platform based on artificial intelligence provided by the present invention has the following beneficial effects:

[0069] The present invention provides a hybrid data platform based on artificial intelligence.

[0070] Raw data: It ensures the reliability of the data source and provides the most basic material for subsequent data processing and analysis. It can collect various types of raw data from different channels, enrich the diversity of data, and provide the possibility for more comprehensive analysis and modeling.

[0071] Domain knowledge: Introducing domain knowledge can provide professional guidance for data processing and modeling. For example, in the medical field, doctors’ expertise can help the platform better understand the meaning and value of medical data and improve the accuracy of disease diagnosis and treatment plan recommendations.

[0072] Data processing: The direction and methods of data processing are clarified. By integrating various data processing technologies, such as cleaning, conversion, and normalization, the quality and availability of data can be improved.

[0073] Algorithms and data modeling: It provides a rich selection of algorithms and modeling methods, and can select the most appropriate algorithms and models according to different problems and data characteristics, thus improving the accuracy and generalization ability of the models;

[0074] Data management: It realizes efficient organization and management of data, including data storage, retrieval, and update, which can ensure the security and integrity of data and prevent data loss and damage;

[0075] Hierarchical integration: effectively integrates data and information at different levels, achieving seamless connection from raw data to final analysis results, avoiding data loss and errors during transmission and conversion between different levels, and improving data consistency and accuracy;

[0076] Data collection: It can collect data from a variety of data sources, including sensors, databases, file systems, networks, etc., enriching the sources and types of data. It can collect data in real time or regularly to ensure the timeliness and freshness of the data.

[0077] Data preprocessing: Through data cleaning, conversion, normalization and other processing, noise and outliers in the data can be removed, the quality and availability of the data can be improved, and data in different formats can be converted into a unified format to facilitate subsequent analysis and processing;

[0078] Data transmission: ensure fast and secure data transmission between different modules and systems, and use encryption and data compression technologies to improve the efficiency and security of data transmission;

[0079] Data aggregation: Integrate and summarize scattered data to form more valuable information. Through data fusion, association analysis and other technologies, the potential relationships and patterns between data can be discovered;

[0080] Data storage: It provides reliable data storage solutions, including distributed storage, cloud storage, etc., which can ensure the security and persistence of data and prevent data loss and damage;

[0081] Data analysis: Using various data analysis techniques, such as statistical analysis, machine learning, and deep learning, to extract valuable information and knowledge from data, we can discover potential patterns and trends in the data and provide support for decision-making;

[0082] Data interpretation: Explain and illustrate the data analysis results so that users can understand the meaning and value of the data, which can help users better use the data analysis results to make decisions and improve the scientificity and rationality of decisions.

[0083] Second embodiment

[0084] Please refer to Figure 2 Based on the hybrid data platform based on artificial intelligence provided by the first embodiment of the present application, the second embodiment of the present application proposes another hybrid data platform based on artificial intelligence. The second embodiment is only a preferred method of the first embodiment, and the implementation of the second embodiment will not affect the independent implementation of the first embodiment.

[0085] Specifically, the second embodiment of the present application provides an artificial intelligence-based hybrid data platform, which is different in that it is an artificial intelligence-based hybrid data platform, wherein the data preprocessing includes data cleaning, data conversion, data dimensionality reduction and data integration, the data cleaning includes processing missing values ​​and processing outliers, the data conversion includes standardization, normalization and logarithmic transformation, the data dimensionality reduction includes principal component analysis and singular value decomposition, and the data integration includes entity recognition and data conflict resolution.

[0086] The missing value processing method includes deletion method and filling method, and the filling method is through mean filling, median filling and most frequent value filling.

[0087] The processing of outliers includes a capping method and a box plot method, wherein the box plot method utilizes a box plot to identify outliers.

[0088] Deletion method: If the proportion of missing values ​​in the data set is small, you can directly delete the records containing missing values. However, this method may lead to information loss, especially when there are many missing values.

[0089] Filling method:

[0090] Mean filling: For numeric data, the missing values ​​can be filled with the average value of the attribute. For example, in a set of students' test score data, if a student's math score is missing, it can be filled with the average of other students' math scores.

[0091] Median filling: For data with outliers, the median can better represent the central trend of the data than the mean. In this case, the median can be used to fill in missing values.

[0092] Most frequent value filling: For categorical data, missing values ​​can be filled with the most frequent category in the attribute.

[0093] Cap method: Set an upper limit and a lower limit, and replace the data outside this range with the upper or lower limit value.

[0094] Box plot method: Use box plot to identify outliers. The upper and lower edges of the box plot represent the upper and lower quartiles of the data, respectively. Data beyond a certain range of the upper and lower edges are considered outliers. These outliers can be deleted, replaced, or analyzed separately according to the specific situation.

[0095] Logarithmic transformation: For some right-skewed data, logarithmic transformation can be performed to make it closer to normal distribution.

[0096] Principal component analysis is a commonly used linear dimensionality reduction method that reduces the dimension of data by finding the main components of the data (i.e., principal components). Principal components are linear combinations of the original data that can retain the variance of the data to the greatest extent.

[0097] The specific steps include: calculating the covariance matrix of the data, performing eigenvalue decomposition on the covariance matrix, selecting the eigenvectors corresponding to the largest eigenvalues ​​as principal components, and projecting the original data onto the principal components to obtain the reduced-dimensional data.

[0098] Singular value decomposition is also an effective dimensionality reduction method. It can decompose a matrix into the product of three matrices to extract the main features of the data.

[0099] The specific steps include: performing singular value decomposition on the original matrix, selecting left singular vectors and right singular vectors corresponding to the first few larger singular values, and projecting the original data onto these vectors to obtain the data after dimensionality reduction.

[0100] Entity Recognition: In multiple data sources, there may be different representations of the same entity.

[0101] Data conflict resolution: When data conflicts exist in multiple data sources, certain strategies need to be adopted to resolve the conflicts.

[0102] Third embodiment

[0103] Please refer to Figure 3 Based on the hybrid data platform based on artificial intelligence provided by the first embodiment of the present application, the third embodiment of the present application proposes another hybrid data platform based on artificial intelligence. The third embodiment is only a preferred method of the first embodiment, and the implementation of the third embodiment will not affect the independent implementation of the first embodiment.

[0104] Specifically, the difference of the third embodiment of the present application is that, in a hybrid data platform based on artificial intelligence, the data modeling includes model selection, data partitioning, hyperparameter adjustment, model integration and regularization, the model selection includes considering the problem type and evaluating the model performance, the data partitioning includes training, validation and test sets and cross-validation, and the hyperparameter adjustment includes grid search, random search and Bayesian optimization.

[0105] Evaluating Model Performance

[0106] Accuracy: For classification problems, accuracy is the ratio of the number of correctly classified samples to the total number of samples. For example, in a classification task with 100 samples, if the model correctly classifies 90 samples, then the accuracy is 90%.

[0107] Precision and recall: Precision refers to the proportion of true positive samples among all samples predicted as positive; recall refers to the proportion of samples correctly predicted as positive among all actual positive samples. For example, in disease diagnosis, a high precision means that the model rarely misclassifies healthy people as patients, and a high recall means that the model can find as many true patients as possible.

[0108] Mean Squared Error (MSE): For regression problems, MSE is the average of the squares of the differences between the predicted values ​​and the true values. The smaller the MSE, the better the prediction effect of the model.

[0109] R 2 Coefficient: The R2 coefficient measures how well the model fits the data, and its value range is between 0 and 1. The closer R2 is to 1, the better the model fits the data.

[0110] The types of problems considered include classification problems, regression problems and clustering problems.

[0111] Classification problem: If the task is to classify data into different categories, you can choose classification models such as logistic regression, decision tree, support vector machine, random forest, neural network, etc.

[0112] For example, in spam classification, you can use logistic regression or random forest models to determine whether an email is spam or not.

[0113] Regression problem: For tasks of predicting continuous values, such as house price prediction, sales prediction, etc., you can choose regression models such as linear regression, polynomial regression, decision tree regression, neural network, etc.

[0114] Clustering problem: When you need to divide the data into different groups, you can use algorithms such as K-Means clustering and hierarchical clustering.

[0115] For example, in customer segmentation, clustering algorithms can be used to divide customers into different groups in order to develop targeted marketing strategies.

[0116] The regression problem is used for the task of predicting continuous values.

[0117] Training, validation, and test sets: It is a common practice in model training to divide the dataset into training, validation, and test sets. The training set is used to train the model, the validation set is used to adjust the model's hyperparameters and evaluate the model's performance, and the test set is used to finally evaluate the model's generalization ability.

[0118] For example, the dataset can be divided into training set, validation set and test set in the proportion of 70%, 20% and 10%. During the training process, the parameters of the model are constantly adjusted to achieve the best performance on the validation set. Finally, the test set is used to evaluate the performance of the model on unseen data.

[0119] Cross-validation: K-fold cross-validation is a commonly used cross-validation method. It divides the data set into K subsets of equal size, takes one of the subsets as the validation set, and the remaining K-1 subsets as the training set, performs K training and validation, and finally takes the average of the K results as the performance evaluation of the model.

[0120] For example, for a data set containing 100 samples, a 5-fold cross validation is performed. The data set is divided into 5 subsets of size 20, and one subset is selected as the validation set each time, and the remaining 4 subsets are used as training sets for training and validation. This is repeated 5 times, and finally 5 performance indicators are obtained, and their average value is taken as the final evaluation result of the model.

[0121] Grid search is an exhaustive search method that iterates over given hyperparameter combinations and finds the one that performs best on a validation set.

[0122] For example, for a support vector machine model, the hyperparameters to be adjusted are the kernel function type (linear, polynomial, radial basis function, etc.) and the regularization parameter C. You can set a hyperparameter range and then perform a grid search on different hyperparameter combinations to find the best hyperparameter combination.

[0123] Random search is similar to grid search, but instead of exhaustively searching all hyperparameter combinations, it randomly samples a certain number of hyperparameter combinations within a given hyperparameter range for testing. Since random search does not need to traverse all hyperparameter combinations, it is more efficient when dealing with high-dimensional hyperparameter spaces.

[0124] Bayesian optimization is a hyperparameter optimization method based on probability models. It establishes a probability model about hyperparameters and model performance, and uses existing test results to update the model to guide the next hyperparameter selection. Bayesian optimization can usually find a better hyperparameter combination in a smaller number of trials.

[0125] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A hybrid data platform based on artificial intelligence, characterized in that: include: A hybrid data platform, the hybrid data platform comprising an ontology level module, a logic level module, an analysis level module, an operation level module, an interpretation level module and a language level module; The ontology level modules are used for raw data, domain knowledge, data processing, algorithms and data modeling; The logic level module is used for data management; The analysis level module is used for level integration; The operational level modules are used for data collection, data preprocessing, data transmission, data aggregation, data storage, data analysis and data interpretation.

2. The artificial intelligence-based hybrid data platform according to claim 1, characterized in that: The language level modules form a case-based grammar for the attributes of various ontological concepts.

3. The artificial intelligence-based hybrid data platform according to claim 1, characterized in that: The explanation level module is used to generate reasonable explanations based on the causal dependencies and their logical representations at the ontology, logic and analysis levels, as well as the data processing results at the operation level.

4. The artificial intelligence-based hybrid data platform according to claim 1, characterized in that: The logical level module references concepts and individuals of the ontology level module by specifying heuristic rules that control the execution of data management operations.

5. The artificial intelligence-based hybrid data platform according to claim 1, characterized in that: The data preprocessing includes data cleaning, data conversion, data dimensionality reduction and data integration. The data cleaning includes processing missing values ​​and processing outliers. The data conversion includes standardization, normalization and logarithmic transformation. The data dimensionality reduction includes principal component analysis and singular value decomposition. The data integration includes entity recognition and data conflict resolution.

6. The artificial intelligence-based hybrid data platform according to claim 5, characterized in that: The missing value processing method includes deletion method and filling method, and the filling method is through mean filling, median filling and most frequent value filling.

7. The artificial intelligence-based hybrid data platform according to claim 5, characterized in that: The processing of outliers includes a capping method and a box plot method, wherein the box plot method utilizes a box plot to identify outliers.

8. The artificial intelligence-based hybrid data platform according to claim 1, characterized in that: The data modeling includes model selection, data partitioning, hyperparameter adjustment, model integration and regularization, the model selection includes considering the problem type and evaluating model performance, the data partitioning includes training, validation and test sets and cross-validation, and the hyperparameter adjustment includes grid search, random search and Bayesian optimization.

9. The artificial intelligence-based hybrid data platform according to claim 8, characterized in that: The types of problems considered include classification problems, regression problems and clustering problems.

10. The artificial intelligence-based hybrid data platform according to claim 9, characterized in that: The regression problem is used for the task of predicting continuous values.

Images