Big data visual modeling method, device and equipment based on machine learning

Abstract

The invention discloses a big data visualization modeling method, device and equipment based on machine learning, and relates to the technical field of big data modeling, the method comprises the following steps: accessing an external data source to an initialized multi-source data directory, and registering external data in the multi-source data directory to a public data directory; training the established initial machine learning model by using the data in the registered public data directory to obtain a target machine learning model; constructing a visual big data model based on the data in the registered public data directory, a data processing tool operator and the target machine learning model; and performing preview debugging on the visual big data model to determine operator parameters of the visual big data model. According to the method, the operator parameters of the model are determined by previewing and debugging the visual big data model, the model prediction result can be previewed, and data from different sources and formats can be processed by supporting access of a multi-source data directory.

Description

Technical Field

[0001] The present application relates to the technical field of big data modeling, and more specifically, to a big data visualization modeling method, device and equipment based on machine learning. Background Art

[0002] With the rapid development of the Internet and big data, more organizations and enterprises are facing the challenge of processing and analyzing massive amounts of data. In order to fully exploit the value and potential of big data, organizations and enterprises need a more intuitive and efficient way to analyze and model big data. Big data analysis and modeling generally require big data-related technicians to write data analysis code, model code, or use professional data processing tools, which requires a high level of big data technicians in the enterprise, making it more difficult for enterprises to carry out data processing and modeling.

[0003] Existing big data modeling technology is mainly based on Spark offline computing, using Spark's built-in model operators for data modeling. The modeling process requires technicians to manually preprocess data and select appropriate model operators to model and mine the data. Subsequently, it is necessary to repeatedly manually analyze the data analysis results and adjust the model parameters.

[0004] However, the process of using Spark's built-in model operators for data modeling is mostly the process of building Spark offline calculations, which not only has low calculation efficiency, but also technical analysts cannot preview the data and model prediction results of each preprocessing link in real time. At the same time, the data model cannot be scheduled in time, and during the data analysis and data modeling process, the types of data warehouses that can be supported are limited. Summary of the invention

[0005] In response to at least one defect or improvement need in the prior art, the present invention provides a big data visualization modeling method, device and equipment based on machine learning, which are used to solve the problems in the prior art of low efficiency in building spark offline computing, and technical analysts are unable to preview the data and model prediction results of each preprocessing link in real time, so the data model cannot be scheduled in time, and in the process of data analysis and data modeling, the types of data warehouses that can be supported are limited.

[0006] To achieve the above object, according to a first aspect of the present invention, a big data visualization modeling method based on machine learning is provided, comprising:

[0007] Connect external data sources to the initialized multi-source data directory, and register external data in the multi-source data directory into the public data directory;

[0008] Using the data in the registered public data directory to train the established initial machine learning model to obtain the target machine learning model;

[0009] Build a visual big data model based on the data in the registered public data catalog, data processing tool operators, and target machine learning models;

[0010] Preview and debug the visualization big data model to determine the operator parameters of the visualization big data model.

[0011] In a possible implementation, connecting the external data source to the initialized multi-source data directory, registering the external data in the multi-source data directory into the public data directory, further includes:

[0012] Connect external data sources to the multi-source data directory according to data attributes, and verify the connected multi-source data directory;

[0013] Determine the newly added data in the multi-source data directory that has passed the verification, and register the newly added data in the public data directory.

[0014] In a possible implementation, using the data in the registered public data directory to train the established initial machine learning model to obtain a target machine learning model also includes:

[0015] Input the data in the registered public data directory into the initial machine learning model to determine the feature columns and label columns;

[0016] Verify the data in the registered public data directory based on feature columns and label columns;

[0017] After verification, adjust the model parameters to train the initial machine learning model and determine the target machine learning model.

[0018] In a possible implementation, the data in the registered public data directory is input into the initial machine learning model to determine the feature columns and label columns, and further includes:

[0019] Preprocess the data in the registered public data directory;

[0020] Preview statistical analysis information of data fields based on preprocessed data in the public data directory;

[0021] Determine the feature columns and label columns of the initial machine learning model based on business modeling requirements and statistical analysis information of the data fields.

[0022] In a possible implementation, building a visualization big data model based on the data in the registered public data catalog, the data processing tool operator and the target machine learning model also includes:

[0023] Pre-process the data in the registered public data directory to determine the valid data resource nodes;

[0024] Connect the data resource nodes with the data processing tool operators using connection lines;

[0025] Connect the target machine learning model with data resource nodes or data processing tool operators to build a visual big data model.

[0026] In a possible implementation, preprocessing the data in the registered public data directory to determine a valid data resource node also includes:

[0027] Determine the root data resource node from the data in the registered public data directory;

[0028] Preprocessing the root data resource node to obtain a preview data resource node;

[0029] Determine valid data from the preview data resource node to obtain a valid data resource node.

[0030] In a possible implementation, previewing and debugging the visualization big data model to determine operator parameters of the visualization big data model further includes:

[0031] Configure initial tool operator parameters according to the data processing tool operator, and configure initial machine learning operator parameters according to the target machine learning model;

[0032] Preview the output data of the initial tool operator parameters and initial machine learning operator parameters through the visualization of the big data model;

[0033] If the output data is valid, the initial tool operator parameters and the initial machine learning operator parameters are the target operator parameters;

[0034] If the output data is invalid, reconfigure the initial tool operator parameters and initial machine learning operator parameters.

[0035] According to a second aspect of the present invention, there is also provided a big data visualization modeling device based on machine learning, comprising:

[0036] A data access module, which is configured to access external data sources to the initialized multi-source data directory and register external data in the multi-source data directory into the public data directory;

[0037] A machine learning module, which is configured to train the established initial machine learning model using the data in the registered public data directory to obtain a target machine learning model;

[0038] A model building module, which is configured to build a visual big data model based on the data in the registered public data catalog, the data processing tool operator and the target machine learning model;

[0039] The parameter configuration module is configured to preview and debug the visualization big data model to determine the operator parameters of the visualization big data model.

[0040] According to the third aspect of the present invention, there is also provided a big data visualization modeling device based on machine learning, which includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs any step of the above-mentioned big data visualization modeling method based on machine learning.

[0041] According to a fourth aspect of the present invention, there is also provided a storage medium storing a computer program executable by an access authentication device, wherein when the computer program runs on the access authentication device, the access authentication device executes any of the steps of the above-mentioned big data visualization modeling method based on machine learning.

[0042] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0043] The present invention provides a big data visualization modeling method based on machine learning, which connects external data sources to an initialized multi-source data directory and registers these data in a public data directory, so that data from different sources can be processed and analyzed under a unified framework, achieving data diversity and comprehensiveness, thereby being able to meet a wider range of data analysis needs. A visualization big data model is constructed based on the data in the registered public data directory, the data processing tool operator, and the target machine learning model. The operator parameters of the model are determined by previewing and debugging the visualization big data model. There is no need for technicians to manually pre-process the data, which improves the computational efficiency of the model. Complex data relationships and patterns can be intuitively displayed in a specific form. The relationship and trend between data makes the data analysis results easier to understand and interpret, so that more accurate and efficient decisions can be made, and the practicality and reliability of the model are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A schematic diagram of a process flow of an embodiment of a big data visualization modeling method based on machine learning provided by the present invention;

[0046] Figure 2The present invention provides Figure 1 A schematic flow chart of an embodiment of step S102;

[0047] Figure 3 The present invention provides Figure 2 A schematic diagram of a flow chart of an embodiment of step S201;

[0048] Figure 4 The present invention provides Figure 1 A schematic flow chart of an embodiment of step S103;

[0049] Figure 5 The present invention provides Figure 4 A schematic diagram of a flow chart of an embodiment of step S401;

[0050] Figure 6 A schematic diagram of the structure of an embodiment of a big data visualization modeling device based on machine learning provided by the present invention;

[0051] Figure 7 A schematic diagram of the structure of a big data visualization modeling device based on machine learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0054] The present invention provides a big data visualization modeling method, device and equipment based on machine learning, which are described below respectively.

[0055] See also Figure 1 , Figure 1A flow chart of an embodiment of a big data visualization modeling method based on machine learning provided by the present invention is provided. In a specific embodiment of the present invention, a big data visualization modeling method based on machine learning is disclosed, including:

[0056] S101, connecting an external data source to an initialized multi-source data directory, and registering external data in the multi-source data directory into a public data directory;

[0057] S102, using the data in the public data directory after registration to train the established initial machine learning model to obtain a target machine learning model;

[0058] S103, building a visual big data model based on the data in the registered public data directory, the data processing tool operator and the target machine learning model;

[0059] S104: Preview and debug the visualized big data model to determine operator parameters of the visualized big data model.

[0060] In the above embodiment, connecting the external data source to the initialized multi-source data directory usually involves connection configuration of the data source, conversion of the data format, and preliminary verification of the data quality. Subsequently, the external data in the multi-source data directory is registered in the public data directory to achieve unified management and access to the data. During the data access process, data cleaning and preprocessing steps can be introduced to remove duplicate data, fill in missing values, convert data types, etc. to ensure the quality and consistency of the data. In addition, data permissions and security policies can be set to control the access rights of different users to the data.

[0061] The initial machine learning model established is trained using the data in the public data directory after registration. The training process may include but is not limited to feature selection, model selection, parameter tuning and other links to obtain the target machine learning model with the best performance. During the model training process, strategies such as cross-validation and grid search can be used to optimize model parameters and improve the generalization ability of the model. At the same time, log information during the training process can be recorded for subsequent analysis and optimization. In addition, the trained model operators can be encapsulated and packaged for use in the visual modeling process.

[0062] Based on the data in the public data directory after registration, the data processing tool operators and the target machine learning model, building a visual big data model requires data flow definition, connection between operators and configuration of visualization components. In the process of building a visualization model, auxiliary functions such as drag-and-drop interface and template library can be introduced to reduce the difficulty of use for users. At the same time, it can support a variety of data processing tool operators, such as data filtering, aggregation, conversion, etc., to meet the data processing needs in different scenarios. In addition, it can also provide a rich library of visualization components, such as line charts, bar charts, scatter plots, etc., so that users can choose the appropriate visualization method according to their needs.

[0063] Preview and debug the visualization big data model to determine the operator parameters of the visualization big data model, including but not limited to data source verification, operator parameter adjustment, and visualization effect optimization. In the process of model preview and debugging, a real-time preview function can be introduced so that users can view the output results of the model at any time. At the same time, a parameter adjustment interface or tool can be provided so that users can adjust operator parameters as needed. In addition, an error detection and feedback mechanism can be introduced to timely discover and solve problems during the model operation process.

[0064] Compared with the prior art, the present embodiment provides a big data visualization modeling method based on machine learning. By connecting external data sources to the initialized multi-source data directory and registering these data in the public data directory, data from different sources can be processed and analyzed under a unified framework, achieving data diversity and comprehensiveness, thereby being able to meet a wider range of data analysis needs. A visualization big data model is constructed based on the data in the registered public data directory, the data processing tool operator, and the target machine learning model. The operator parameters of the model are determined by previewing and debugging the visualization big data model. There is no need for technicians to manually pre-process the data, which improves the computational efficiency of the model. Complex data relationships and patterns can be intuitively displayed in a specific form. The relationship and trend between data makes the data analysis results easier to understand and interpret, so that more accurate and efficient decisions can be made, and the practicality and reliability of the model are improved.

[0065] In some embodiments of the present invention, connecting the external data source to the initialized multi-source data directory and registering the external data in the multi-source data directory into the public data directory further includes:

[0066] Connect external data sources to the multi-source data directory according to data attributes, and verify the connected multi-source data directory;

[0067] Determine the newly added data in the multi-source data directory that has passed the inspection, and register the newly added data in the public data directory.

[0068] In the above embodiment, first, the external data source is carefully classified according to its data attributes, such as data type (structured, semi-structured, unstructured), data source (public API, private database, third-party service, etc.), data format (CSV, JSON, XML, etc.), and data purpose (analysis, reporting, decision support, etc.). Classification is helpful for subsequent data management, query and optimization processing. Then, according to the classification results, the external data source is connected to the initialized multi-source data directory. As a centralized storage and indexing system, this directory can accommodate data from different sources with different characteristics.

[0069] A default Doris data multi-source directory is set in the present invention, which can support various types of external data sources, register with Doris in the form of a multi-source directory, and use Doris's analysis and calculation engine to perform data query and analysis. The supported external data source types include common relational databases such as MySQL, Oracle, and Dameng databases, and common non-relational databases such as Hive and Elasticsearch. It can also further support the access of other data sources that meet certain specifications and are not listed here. On this basis, the data tables in the external data source and the built-in data source can be registered in the public data directory in the data management module, and the directory structure can be customized according to business needs. The public data directory resources provide basic data support for the upper-level modules, and can provide data query analysis, modification and update to the upper layer in the form of an application program interface.

[0070] After access, in order to ensure the accuracy, completeness and consistency of the data, a comprehensive inspection of the accessed multi-source data directory is carried out. The inspection process may involve verification of the data format, inspection of data integrity (such as ensuring that there are no missing values ​​or duplicate records), and assessment of data quality (such as checking whether the data conforms to the expected value range and whether the logical relationship between the data is established), etc., which can effectively avoid errors or deviations in subsequent data processing and analysis.

[0071] Once the data in the multi-source data directory passes the inspection, it will automatically identify and mark the new data, which represents the latest market information, user behavior data, business transaction records, etc. Next, this part of the new data will be registered in the public data directory. As a unified data access portal, the public data directory provides standardized data descriptions, metadata information and access rights management, allowing authorized users or systems to easily discover, understand and use these data resources.

[0072] See also Figure 2 , Figure 2 The present invention provides Figure 1In some embodiments of the present invention, the initial machine learning model established is trained using the data in the public data directory after registration to obtain the target machine learning model, and further includes:

[0073] S201, inputting the data in the registered public data directory into the initial machine learning model to determine feature columns and label columns;

[0074] S202, verifying the data in the registered public data directory based on the feature column and the label column;

[0075] S203. After verification, adjust the model parameters to train the initial machine learning model and determine the target machine learning model.

[0076] In the above embodiment, the data in the public data directory after registration is used as training material and input into the established initial machine learning model. In order to ensure the validity and relevance of the data, feature columns and label columns that are highly correlated with the target variable are selected from the public data directory according to the specific needs of the problem. The feature column contains key information for prediction or classification, while the label column represents the target value that needs to be predicted or classified.

[0077] In order to ensure the quality of input data, these feature columns and label columns are strictly checked. The verification process may include but is not limited to data cleaning (such as removing duplicate values ​​and processing missing values), data conversion (such as data standardization and normalization), and data verification (such as checking data format, value range, logical consistency, etc.). These verification processes are crucial to eliminating data noise and improving data quality, and can significantly improve the training efficiency and prediction accuracy of the model.

[0078] Once the data has passed the verification, we select the appropriate machine learning algorithm (such as linear regression, decision tree, random forest, neural network, etc.) according to the characteristics of the data and the complexity of the problem, and train the initial machine learning model based on the feature column and label column. During the training process, we continuously adjust the model parameters (such as learning rate, number of iterations, regularization coefficient, etc.) to find the optimal model configuration so that the model performs best on the training set.

[0079] Finally, when the performance of the model on the training set tends to be stable, and the performance on the validation set also meets or exceeds expectations, the trained model can be determined as the target machine learning model. The target machine learning model not only contains the structure and parameters of the model, but also encapsulates the knowledge and experience of the entire process such as data preprocessing, feature selection, and model training, which can be easily applied to new data for prediction or classification tasks.

[0080] It should be noted that the machine learning module supports the training of common machine learning models, including classification algorithms such as decision tree classification, clustering algorithms such as K-means clustering, prediction algorithms such as linear regression prediction, and association rule analysis such as frequent pattern growth rules. In machine learning, the trained model is stored in the object storage service system or distributed file system in the form of a file. Furthermore, the model results can be displayed in the machine learning module. Different types of models have different display forms and contents. The displayed content includes but is not limited to mathematical formulas, model training parameters and result indicator tables and graphs, tree diagrams, and confusion matrix tables. Among them, the model result indicators include the number of clusters, cluster square value summary, model recall rate, accuracy rate, classification tree diagram, classification confusion matrix, etc. There are many model result indicators, which cannot be enumerated based on each model, but most of the model result indicators are based on the model training result data of Spark machine learning.

[0081] In a preferred embodiment of the present invention, the model training process uses a data processing engine that combines Spark and Doris, and the trained model files will be stored in a file storage system that supports a variety of object storage service systems such as Minio, OSS, and distributed storage systems such as HDFS. The result parameters of the model training can be stored in the metadata MySQL of this system.

[0082] The model results generated by machine learning are used in the subsequent process in the form of machine learning model operators. The specific usage method is to drag the machine learning model operator to the canvas, select the successfully trained model in the model selection drop-down box, and finally configure the machine learning operator parameters.

[0083] See also Figure 3 , Figure 3 The present invention provides Figure 2 In some embodiments of the present invention, the data in the registered public data directory is input into the initial machine learning model to determine the feature column and the label column, and further includes:

[0084] S301, preprocessing the data in the public data directory after registration;

[0085] S302, previewing statistical analysis information of data fields according to the preprocessed data in the public data directory;

[0086] S303. Determine the feature columns and label columns of the initial machine learning model based on business modeling requirements and statistical analysis information of the data fields.

[0087] In the above embodiment, a series of preprocessing operations are performed on the data in the public data directory after registration in order to eliminate noise, inconsistency and missing values ​​in the data and ensure the quality and integrity of the data. Preprocessing may include data cleaning (such as deleting duplicate records, filling or deleting missing values), data conversion (such as data type conversion, data encoding), data standardization or normalization (to ensure that different features are on the same scale), etc. These preprocessing operations can significantly improve data quality and lay a solid foundation for subsequent feature selection and model training.

[0088] Next, through the application interface of the data resource module, the statistical analysis information of the data field can be previewed through the data in the preprocessed public data directory. The statistical analysis information of the data field can be judged through the preview to see whether it is normal. The statistical analysis information may include the mean, standard deviation, maximum value, minimum value, quartiles, frequency distribution, correlation matrix, etc. This information not only helps to identify potential problems and anomalies in the data, but also can be used for subsequent feature selection and label column determination.

[0089] The feature column is the input variable used for prediction or classification in the model, while the label column is the target variable that needs to be predicted or classified. Based on the business modeling requirements and the statistical analysis information of the data field, when determining the feature column, the statistical characteristics of the data (such as variance, correlation), business logic (such as which factors may affect the target variable) and the complexity of the model (avoid overfitting caused by too many features) are considered. At the same time, according to the statistical analysis information, feature columns with high correlation with the target variable, large amount of information and independent of each other are selected. When determining the label column, data fields that can clearly reflect the prediction or classification target will be selected, and their data quality will be ensured to be reliable and the values ​​are clear. Business modeling requirements are necessary conditions that need to be met in actual operations, and can be adjusted and set according to specific circumstances. The present invention does not impose further restrictions on this.

[0090] See also Figure 4 , Figure 4 The present invention provides Figure 1 In some embodiments of the present invention, the process of building a visualization big data model based on the data in the registered public data directory, the data processing tool operator and the target machine learning model also includes:

[0091] S401, pre-processing the data in the registered public data directory to determine valid data resource nodes;

[0092] S402, connecting the data resource node with the data processing tool operator using a connection line;

[0093] S403. Connect the target machine learning model with the data resource node or the data processing tool operator to build a visual big data model.

[0094] In the above embodiment, the data in the public data directory after registration is preprocessed to determine valid data resource nodes. The preprocessing may include but is not limited to data cleaning (such as deleting invalid records and processing missing values), data conversion (such as data type conversion and data format unification), and data verification (such as data integrity check and data consistency verification). Through these preprocessing operations, the quality of the data can be ensured, and valid data resource nodes can be identified, providing a reliable data source for subsequent data processing and model building.

[0095] Next, use the connection line to connect the data resource node with the data processing tool operator. The data processing tool operator is a series of predefined algorithms or functions used to perform specific data processing tasks. These operators may include data filtering, data aggregation, data conversion, data sorting, etc. By connecting the data resource node with the corresponding data processing tool operator, a series of processing operations can be performed on the data to meet the requirements of the target machine learning model for data format and quality.

[0096] After data processing is completed, the operators of the target machine learning model are connected to the data resource nodes or data processing tool operators to build a visual big data model. The target machine learning model is a machine learning algorithm or model selected according to the specific application scenario and prediction target. These operators can include classification models, regression models, clustering models, etc. When building a model, it is necessary to ensure that the data format and quality of the model input match the model requirements, and to display and operate intuitively through a visual interface.

[0097] In the process of building a visual big data model, it is also necessary to focus on model optimization and verification. By performing operations such as model parameter tuning and cross-validation, the model's prediction accuracy and generalization ability can be improved. At the same time, through the tools and functions provided by the visualization interface, the model's training process, performance evaluation results, and prediction results can be intuitively viewed, so as to better understand and optimize the model.

[0098] See also Figure 5 , Figure 5 The present invention provides Figure 4 In some embodiments of the present invention, preprocessing the data in the registered public data directory to determine the valid data resource node also includes:

[0099] S501, determining the root data resource node from the data in the public data directory after registration;

[0100] S502, pre-processing the root data resource node to obtain a preview data resource node;

[0101] S503: Determine valid data from the preview data resource nodes to obtain valid data resource nodes.

[0102] In the above embodiment, first, the root data resource nodes are determined from the data in the public data directory after registration. These root data resource nodes are basic data units in the public data directory, which may include data sources in various forms such as database tables, data files, data interfaces, etc. When determining the root data resource nodes, the public data directory needs to be comprehensively sorted and classified to ensure that all relevant data resources are taken into consideration.

[0103] Next, the root data resource node is preprocessed to obtain the preview data resource node. The preprocessing steps may include but are not limited to data cleaning (such as deleting invalid records and processing missing values), data conversion (such as data type conversion and data format unification), and data verification (such as data integrity check and data consistency verification). Through these preprocessing operations, we can have a preliminary understanding of the structure and content of the data, providing a basis for subsequent data analysis and processing.

[0104] After obtaining the preview data resource nodes, further determining valid data from these data to obtain valid data resource nodes may require in-depth analysis and evaluation of the data based on business logic, data quality assessment standards, and the needs of the target machine learning model. This ensures that only high-quality data that is highly relevant to the prediction target is retained as the basis for subsequent model training and prediction.

[0105] After obtaining valid data resource nodes, it is necessary to further process and analyze these data to meet the requirements of the target machine learning model for data format and quality, which may include data normalization, standardization, feature selection, feature extraction and other operations. Through these operations, the quality and availability of data can be further improved, providing strong support for model construction and optimization.

[0106] In some embodiments of the present invention, previewing and debugging the visualization big data model to determine operator parameters of the visualization big data model further includes:

[0107] Configure initial tool operator parameters according to the data processing tool operator, and configure initial machine learning operator parameters according to the target machine learning model;

[0108] Preview the output data of the initial tool operator parameters and initial machine learning operator parameters through the visualization of the big data model;

[0109] If the output data is valid, the initial tool operator parameters and the initial machine learning operator parameters are the target operator parameters;

[0110] If the output data is invalid, reconfigure the initial tool operator parameters and initial machine learning operator parameters.

[0111] In the above embodiment, first, according to the selected data processing tools (such as ETL tools, data cleaning tools, etc.), according to the characteristics of the data and preprocessing requirements, the initial tool operator parameters are configured, and these parameters include but are not limited to the connection information of the data source, data screening conditions, data conversion rules, etc. Then, for the target machine learning model (such as classification model, regression model, clustering model, etc.), according to its algorithm principle and model training requirements, the initial machine learning operator parameters are configured, and these parameters may include learning rate, number of iterations, regularization coefficient, feature selection strategy, etc.

[0112] After configuring the initial operator parameters, use the preview function provided by the visualization big data model platform to preview the output data of the initial tool operator parameters and the initial machine learning operator parameters, so that you can visually check the shape and distribution of the processed data, as well as the model's fit to the data. The preview content includes the intermediate results of each stage of the data flow, the format of the data after conversion, and the preliminary results of the model prediction.

[0113] Next, the previewed output data is evaluated for validity, which requires checking the integrity, accuracy, consistency, etc. of the data, as well as the accuracy and stability of the model prediction results. If the output data is valid, that is, it meets the expected data processing and model prediction requirements, it means that the initial tool operator parameters and initial machine learning operator parameters are appropriate and can be determined as the final target operator parameters.

[0114] If the output data is invalid, that is, there are problems such as data anomalies and inaccurate model prediction results, the initial tool operator parameters and initial machine learning operator parameters need to be reconfigured. After reconfiguring the parameters, preview the visual big data model and evaluate the validity of the output data again until the optimal parameter combination is found to ensure that the model can accurately and efficiently process data and produce valuable analysis results.

[0115] It is understandable that whether the data is valid requires an integrity check on the data, and comparison with a set threshold, quality assessment, etc., which are all relatively conventional existing technologies and can be directly used in the present invention, and no further elaboration is made on this.

[0116] In order to better implement the big data visualization modeling method based on machine learning in the embodiment of the present invention, on the basis of the big data visualization modeling method based on machine learning, correspondingly, please refer to Figure 6 , Figure 6This is a schematic diagram of the structure of an embodiment of a big data visualization modeling device based on machine learning provided by the present invention. The embodiment of the present invention provides a big data visualization modeling device 600 based on machine learning, including:

[0117] A data access module 610 is configured to access an external data source to the initialized multi-source data directory and register the external data in the multi-source data directory into the public data directory;

[0118] A machine learning module 620 is configured to train the established initial machine learning model using the data in the registered public data directory to obtain a target machine learning model;

[0119] A model building module 630, which is configured to build a visual big data model based on the data in the registered public data catalog, the data processing tool operator and the target machine learning model;

[0120] The parameter configuration module 640 is configured to preview and debug the visualization big data model to determine the operator parameters of the visualization big data model.

[0121] It should be noted here that: the device 600 provided in the above embodiment can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above method embodiments, which will not be repeated here.

[0122] See also Figure 7 , Figure 7 A schematic diagram of the structure of a big data visualization modeling device based on machine learning provided by an embodiment of the present invention. Based on the above-mentioned big data visualization modeling method based on machine learning, the present invention also provides a big data visualization modeling device based on machine learning. The big data visualization modeling device based on machine learning can be a computing device such as a mobile terminal, a desktop computer, a notebook, a PDA, and a server. The big data visualization modeling device based on machine learning 700 includes a processor 710, a memory 720, and a display 730. Figure 7 Only some components of the synchronous tracking flying welding equipment for real-time measurement of battery height are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0123] In some embodiments, the memory 720 may be an internal storage unit of the big data visualization modeling device 700 based on machine learning, such as a hard disk or memory of the big data visualization modeling device 700 based on machine learning. In other embodiments, the memory 720 may also be an external storage device of the big data visualization modeling device 700 based on machine learning, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the big data visualization modeling device 700 based on machine learning. Further, the memory 720 may also include both an internal storage unit of the big data visualization modeling device 700 based on machine learning and an external storage device. The memory 720 is used to store application software and various types of data installed in the big data visualization modeling device 700 based on machine learning, such as the program code for installing the big data visualization modeling device 700 based on machine learning. The memory 720 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a big data visualization modeling program 740 based on machine learning is stored in the memory 720, and the big data visualization modeling program 740 based on machine learning can be executed by the processor 710, thereby realizing the big data visualization modeling method based on machine learning of each embodiment of the present application.

[0124] In some embodiments, the processor 710 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 720 or process data, such as executing a big data visualization modeling method based on machine learning.

[0125] In some embodiments, the display 730 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 730 is used to display information on the machine learning-based big data visualization modeling device 700 and to display a visualized user interface. The components 710-730 of the machine learning-based big data visualization modeling device 700 communicate with each other via a system bus.

[0126] In one embodiment, when the processor 710 executes the big data visualization modeling program 740 based on machine learning in the memory 720, the steps in the above big data visualization modeling method based on machine learning are implemented.

[0127] This embodiment also provides a computer-readable storage medium, on which a big data visualization modeling program based on machine learning is stored. When the big data visualization modeling program based on machine learning is executed by a processor, the following steps are implemented:

[0128] Connect external data sources to the initialized multi-source data directory, and register external data in the multi-source data directory into the public data directory;

[0129] Using the data in the registered public data directory to train the established initial machine learning model to obtain the target machine learning model;

[0130] Build a visual big data model based on the data in the registered public data catalog, data processing tool operators, and target machine learning models;

[0131] Preview and debug the visualization big data model to determine the operator parameters of the visualization big data model.

[0132] In summary, a big data visualization modeling method based on machine learning connects external data sources to the initialized multi-source data directory and registers these data in the public data directory, so that data from different sources can be processed and analyzed under a unified framework, achieving data diversity and comprehensiveness, thereby being able to meet a wider range of data analysis needs. A visualization big data model is constructed based on the data in the registered public data directory, the data processing tool operator, and the target machine learning model. The operator parameters of the model are determined by previewing and debugging the visualization big data model. There is no need for technicians to manually pre-process the data, which improves the computational efficiency of the model. Complex data relationships and patterns can be intuitively displayed in a specific form. The relationship and trend between data makes the data analysis results easier to understand and interpret, so that more accurate and efficient decisions can be made, and the practicality and reliability of the model are improved.

[0133] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0134] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0135] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0140] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0141] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A big data visualization modeling method based on machine learning, characterized in that: include: Connecting external data sources to the initialized multi-source data directory, and registering external data in the multi-source data directory into the public data directory; Using the data in the registered public data directory to train the established initial machine learning model to obtain the target machine learning model; Building a visual big data model based on the data in the registered public data directory, the data processing tool operator and the target machine learning model; The visualized big data model is previewed and debugged to determine operator parameters of the visualized big data model.

2. The big data visualization modeling method based on machine learning according to claim 1, characterized in that: The step of connecting the external data source to the initialized multi-source data directory and registering the external data in the multi-source data directory into the public data directory further includes: Connecting the external data source to the multi-source data directory according to data attribute classification, and inspecting the connected multi-source data directory; Determine the newly added data in the multi-source data directory that has passed the verification, and register the newly added data in the public data directory.

3. The big data visualization modeling method based on machine learning according to claim 1, characterized in that: The method of using the data in the registered public data directory to train the established initial machine learning model to obtain the target machine learning model also includes: Input the data in the registered public data directory into the initial machine learning model to determine feature columns and label columns; Verifying the data in the registered public data directory based on the feature column and the label column; After verification, adjust the model parameters to train the initial machine learning model and determine the target machine learning model.

4. The big data visualization modeling method based on machine learning as claimed in claim 3, characterized in that: The step of inputting the data in the registered public data directory into the initial machine learning model to determine the feature columns and label columns further includes: Preprocess the data in the registered public data directory; Preview statistical analysis information of data fields based on preprocessed data in the public data directory; The feature columns and label columns of the initial machine learning model are determined based on business modeling requirements and statistical analysis information of the data fields.

5. The big data visualization modeling method based on machine learning according to claim 1, characterized in that: The constructing of a visualization big data model based on the data in the registered public data directory, the data processing tool operator and the target machine learning model also includes: Preprocessing the data in the registered public data directory to determine valid data resource nodes; Connecting the data resource node with the data processing tool operator using a connection line; Connect the target machine learning model with the data resource node or the data processing tool operator to construct a visual big data model.

6. The big data visualization modeling method based on machine learning according to claim 5, characterized in that: The preprocessing of the data in the registered public data directory to determine a valid data resource node also includes: Determine the root data resource node from the data in the registered public data directory; Preprocessing the root data resource node to obtain a preview data resource node; Determine valid data from the preview data resource node to obtain a valid data resource node.

7. The big data visualization modeling method based on machine learning according to claim 6, characterized in that: The step of previewing and debugging the visualized big data model to determine operator parameters of the visualized big data model further includes: configuring initial tool operator parameters according to the data processing tool operator, and configuring initial machine learning operator parameters according to the target machine learning model; Previewing output data of the initial tool operator parameters and the initial machine learning operator parameters through the visual big data model; If the output data is valid, the initial tool operator parameters and the initial machine learning operator parameters are target operator parameters; If the output data is invalid, reconfigure the initial tool operator parameters and the initial machine learning operator parameters.

8. A big data visualization modeling device based on machine learning, characterized in that: include: A data access module configured to access an external data source to an initialized multi-source data directory and register the external data in the multi-source data directory into a public data directory; A machine learning module, which is configured to train the established initial machine learning model using the data in the registered public data directory to obtain a target machine learning model; A model building module, which is configured to build a visual big data model based on the data in the registered public data directory, the data processing tool operator and the target machine learning model; A parameter configuration module is configured to preview and debug the visualization big data model to determine operator parameters of the visualization big data model.

9. A big data visualization modeling device based on machine learning, characterized in that: It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the big data visualization modeling method based on machine learning as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: It stores a computer program that can be executed by an access authentication device. When the computer program runs on the access authentication device, the access authentication device executes the steps of the big data visualization modeling method based on machine learning as described in any one of claims 1 to 7.

Images