A visual modeling method, device, equipment and medium based on big data

Through the visual modeling method based on big data, the problems of high technical threshold and poor real-time responsiveness of big data analysis are solved, and the hybrid computing of multi-data sources and batch flow integrated computing are realized, which reduces the complexity of big data analysis and is non-technical The staff provides simple and easy-to-use tools.

CN117762396BActive Publication Date: 2024-11-01AEROSPACE SHENZHOU INTELLIGENT SYST TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202311804174.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-26
Publication Date
2024-11-01
Estimated Expiration
2043-12-26

AI Technical Summary

Technical Problem

The existing big data analysis technology has a high threshold, requires collaboration in multiple positions, low work efficiency, poor real-time responsiveness, and most data modeling products only support offline data calculations, and cannot realize hybrid computing for multiple data sources.

Method used

Provide a visual modeling method based on big data. By preprocessing and modeling the data resources in response to user requests, a data model of directed acyclic graph DAG is formed, and parsing it into a model operator to connect it to the data warehouse. Supports batch offline calculation, streaming calculation and batch streaming integration.

Benefits of technology

It lowers the threshold for big data analysis and mining, realizes zero-code visualization operations, supports mixed computing of multiple data sources, improves work efficiency and real-time responsiveness, and provides non-technical personnel with simple and easy-to-use big data mining tools.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117762396B_ABST
    Figure CN117762396B_ABST
Patent Text Reader

Abstract

The present invention provides a visualization modeling method, device, equipment and medium based on big data. It relates to the technical field of big data modeling, so as to reduce the threshold of big data analysis and mining and realize zero-code visualization operation. The method comprises: performing a first preprocessing on data resources, wherein the data resources include a data source and a data warehouse, performing a second preprocessing on the data resources to form a data model of a directed acyclic graph DAG, a back-end system parsing the data model of the directed acyclic graph DAG, converting the parsed data model into a model operator, and connecting the model operator to the data warehouse; submitting the model operator to a big data cluster for data analysis and mining. The visualization modeling device based on big data is applied to a visualization modeling method based on big data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data modeling technology, and in particular to a big data-based visualization modeling method, device, equipment and medium. Background Art

[0002] More and more governments, enterprises and other institutions are beginning to realize that data is becoming the most important asset of organizations, and data analysis capabilities are becoming the core competitiveness of organizations. The "big data" era is a historic opportunity for the transformation of government management. For enterprises, the use of massive data will become the basis for future competition and growth. However, in the actual application of big data analysis, there are some problems. First, big data analysis requires multi-position collaboration and low work efficiency. Due to the high threshold of big data analysis, data analysts are often required to inform technical personnel of the algorithm of the data model, and technical personnel operate the big data platform to conduct data mining and analysis, and synchronize the analysis results to data analysts. Data analysts, based on the analysis results, inform technical personnel of the optimized data model algorithm again, and technical personnel repeat the above work and synchronize the analysis results to data analysts. In this way, a model operator with practical effects is finally formed. Second, most data modeling products currently only support offline data calculation, and the real-time responsiveness is low. At present, most data modeling products mainly model offline data, and the data model algorithm must be scheduled through scheduled tasks, which cannot meet the responsiveness requirements for real-time data models. Third, most data modeling products can only select one data source during the data modeling process and cannot implement mixed calculations of multiple data sources. The data preview of each modeling step cannot respond instantly. Currently, most data modeling products can only support sample data for data preview.

[0003] At present, the technology of big data visualization modeling in China is mainly offline computing, with pre-defined model operators, poor compatibility across big data platforms, single data warehouse support, low overall productization level, and complex system deployment. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a visualization modeling method, device, equipment and medium based on big data to lower the threshold of big data analysis and mining and realize zero-code visualization operation.

[0005] The present invention provides a visualization modeling method based on big data, the method comprising:

[0006] Step S1: in response to a user's data model building request, performing a first preprocessing on data resources, the data resources including a data source and a data warehouse;

[0007] Step S2: in response to the user's data model building request, performing a second preprocessing on the data resources to form a data model of a directed acyclic graph DAG;

[0008] Step S3: In response to the user's data model parsing request, the backend system parses the data model of the directed acyclic graph DAG, converts the parsed data model into a model operator, and connects the model operator to the data warehouse;

[0009] Step S4: In response to the user's computing request, the model operator is submitted to the big data cluster for data analysis and mining.

[0010] Preferably, in step S1, the data resources are first preprocessed, including: data access, data cleaning and data scheduling are performed on the data in multiple data sources, and then the data in the data sources enter the data warehouse.

[0011] Preferably, the second preprocessing of the data resource in step S2 includes:

[0012] Drag the model operator and the first preprocessed data resources onto the visual modeling canvas, form a directed acyclic graph (DAG) data model by orchestrating and configuring the data resources and model operators, define the data protocol, trigger the save operation function, and save the directed acyclic graph (DAG) data model to the business database.

[0013] Preferably, the step S2 further includes: in response to a data model building request from a user, during the data model building process, real-time previewing of the operation results of the data model by calling sampling data in real time.

[0014] Preferably, in response to the user's data model parsing request in step S3, the backend system parses the data model of the directed acyclic graph DAG, including:

[0015] Use Spark-core and SparkSQL to develop operator program templates;

[0016] Encapsulate the operator program template into a running code package Runtime.jar;

[0017] Configure the running code package Runtime.jar in the big data cluster Spark.

[0018] Connect to the big data cluster Spark through the Livy interface and call the running code package Runtime.jar.

[0019] Preferably, the step S3 converts the parsed data model into a model operator and connects the model operator to the data warehouse, including:

[0020] Call the livy interface to pass the model's running configuration parameters to the model running code package Runtime.jar. The model running code package Runtime.jar parses the data model parameters of the directed acyclic graph DAG, obtains the input data warehouse information, model operators, and output data warehouse information, and puts the response information in different lists.

[0021] After the Spark task starts executing, it processes the input data warehouse information, model operators, and output data warehouse information. According to the predefined data warehouse type, it loads the spark connector of the corresponding data warehouse and loads the data into the DataFrame to prepare for the subsequent model operator operation.

[0022] Execute the model operator, convert the model operator information into an ordered set of model operators, loop through the ordered set of model operators, and according to the model operator type, the operator factory creates an operator instance based on the operator name reflection, loads the required operator through reflection, and connects to the actual data warehouse at the same time, executes the operators in sequence to process the data according to the arranged process, and the data input and output between operators are connected in series through DataFrame;

[0023] Execute data output and create a table structure in the corresponding data warehouse according to the output parameters. After the workflow is completed, the result data is output to the result table.

[0024] Preferably, the data sources include Mysql database, Kafka database, Oracle database, Sql server database, Postgresql database, DM database, message middleware and FTP resources; the data warehouse includes Hive database, Guassdb database, Gbase database, TiDB database and Clickhouse.

[0025] Compared with the prior art, the visual modeling method based on big data provided by the present invention has the following beneficial effects: by responding to the user's data model construction request, the data resources are preprocessed for the first time, and in response to the user's data model construction request, the data resources are preprocessed for the second time to form a data model of a directed acyclic graph DAG; in response to the user's data model parsing request, the back-end system parses the data model of the directed acyclic graph DAG, converts the parsed data model into a model operator, and connects the model operator to the data warehouse; in response to the user's calculation request, the model operator is submitted to the big data cluster for data analysis and mining to realize batch offline calculation, streaming calculation and batch-stream integrated calculation. The present invention uses a variety of data resources and model operators through a visual drag-and-drop layout, and through the arrangement, configuration, and debugging of data resources and model operators, the previous offline and paper business knowledge and experience are precipitated into an intuitive and clear digital model, creating a simple and easy-to-use big data mining tool for non-technical personnel, and lowering the threshold for big data application.

[0026] The present invention also provides a visual modeling device based on big data, the device comprising:

[0027] A data resource module, used to respond to a user's data model building request and provide data resources, wherein the data resources include a data source and a data warehouse;

[0028] A data processing module, configured to perform a first preprocessing on data resources in response to a user's data model building request;

[0029] A visual modeling module, configured to perform a second preprocessing on the data resources in response to a data model building request from a user, so as to form a data model of a directed acyclic graph DAG;

[0030] A background processing module, which is used to respond to a user's data model parsing request, and the backend system parses the data model of the directed acyclic graph DAG, converts the parsed data model into a model operator, and connects the model operator to the data warehouse;

[0031] The big data computing module is used to respond to the user's computing request and submit the model operator to the big data cluster for data analysis and mining.

[0032] Compared with the prior art, the beneficial effects of a visual modeling device based on big data provided by the present invention are the same as the beneficial effects of a visual modeling method based on big data described in the above technical solution, which will not be elaborated here.

[0033] The present invention also provides an electronic device, comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, and when the computer program is executed by the processor, the steps in any one of the above-mentioned methods for visual modeling based on big data are implemented.

[0034] Compared with the prior art, the beneficial effects of the electronic device provided by the present invention are the same as the beneficial effects of the big data-based visual modeling method described in the above technical solution, which will not be elaborated here.

[0035] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any one of the above-mentioned visualization modeling methods based on big data are implemented.

[0036] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the big data-based visual modeling method described in the above technical solution, and will not be elaborated here.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a visual modeling method based on big data provided by an embodiment of the present invention is shown;

[0040] Figure 2 A schematic diagram of operation scheduling of a visual modeling method based on big data provided by an embodiment of the present invention is shown;

[0041] Figure 3 A schematic structural diagram of a visual modeling device based on big data provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0042] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0043] The "multiple" mentioned in this embodiment refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The words "exemplary" or "for example" are used to indicate examples, illustrations or explanations, and are intended to present related concepts in a specific way, and should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions.

[0044] Before introducing the embodiments of the present application, the following definitions are given for the relevant terms involved in the embodiments of the present application:

[0045] Hadoop: Hadoop is an open source distributed computing and storage framework developed and maintained by the Apache Foundation. Hadoop provides reliable, scalable application-layer computing and storage support for large computer clusters. It allows the use of simple programming models to distribute large data sets across computer clusters, and supports expansion from a single computer to thousands of computers.

[0046] Hive: Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files into a data table and provide SQL query functions.

[0047] Spark: Apache Spark is an open source data analysis cluster computing framework. It is a fast and general computing engine designed for large-scale data processing. It provides a faster and more general data processing platform. Compared with Hadoop, Spark can increase the speed of programs by 100 times when running in memory or 10 times when running on disk, so it is an indispensable software for data analysis.

[0048] Livy: Livy is an open source REST service based on Spark. It can submit code snippets or serialized binary codes to the Spark cluster for execution through REST.

[0049] JAVA SPI: The English name is Service Provider Interface, abbreviated as SPI, which is a built-in standard of Java that allows different developers to implement a specific service.

[0050] XXL_JOB: XXL-JOB is a distributed task scheduling platform, whose core design goal is to develop quickly, learn easily, be lightweight and easy to expand. It is now open source and connected to the online product lines of many companies, ready to use out of the box.

[0051] An embodiment of the present invention provides a visualization modeling method based on big data. Figure 1 A flowchart of a visual modeling method based on big data provided by an embodiment of the present invention is shown, such as Figure 1 As shown, the method includes:

[0052] Step S1: In response to a user's request to build a data model, a first preprocessing is performed on data resources, where the data resources include a data source and a data warehouse.

[0053] It should be noted that in response to the user's request to build a data model, the data resources are first preprocessed. It should be understood that the first preprocessing includes data access, data cleaning and data scheduling of data from multiple data sources before the data source data enters the data warehouse. The data source supports docking with relational databases, message middleware and FTP resources, etc. The docking relational database can be more than 30 databases such as Mysql database, Kafka database, Oracle database, Sql server database, Postgresql database, DM database, etc.; the data warehouse supports common databases such as Hive database, Guassdb database, Gbase database, TiDB database and Clickhouse.

[0054] For data access from multiple data sources, in order to realize data source plug-in access, JAVA SPI and JAVA custom class loader classloader can be used to realize data access and metadata information viewing. The specific implementation method is to define the top-level common interface, such as the interface interface, through which the corresponding data source implements the common interface method to form a plug-in package, and according to the JAVA SPI requirements, place a configuration file named the full path of the interface in META-INF\services of the JAR package, and the file content is the full path of the implementation class. At the same time, develop a custom classloader loader, specify the location of the plug-in package, dynamically load the plug-in package, and effectively avoid conflicts caused by the introduction of multiple plug-ins in the project.

[0055] For data cleaning, the data from the data source is synchronized in batches to Hive or other data warehouses supported by this system through data access, data cleaning and data scheduling. The data is cleaned and processed during the data synchronization process, and the data is managed in layers. The specific implementation method is to use the open source datax data integration tool and carry out secondary development to achieve more complex data cleaning and conversion functions. It should be understood that after synchronizing the data to the data warehouse, the data administrator can configure and authorize the data resources. The authorization methods include users, roles, and organizations to meet the hierarchical and classified use of data resources.

[0056] For data scheduling, the data scheduling tool uses XXL-JOB to implement data scheduling methods such as timing and cycle.

[0057] Step S2: In response to the user's request to build a data model, perform a second preprocessing on the data resources to form a data model of a directed acyclic graph DAG.

[0058] It should be noted that in response to the user's request to build a data model, the model operator and the first preprocessed data resources are dragged onto the visual modeling canvas, and a directed acyclic graph DAG data model is formed through the orchestration and configuration operations of the data resources and the model operator, the data protocol is defined, the save operation function is triggered, and the directed acyclic graph DAG data model is saved to the business database.

[0059] After dragging and dropping the model operator and the first preprocessed data resource on the visual modeling canvas, save the data model of the directed acyclic graph DAG to the business database. The data model of the directed acyclic graph DAG includes data resources, model operators, operating parameters, etc. The data format of the data model of the directed acyclic DAG includes model number, model name, model description, model group number, creator, creation time, model parameter information, model label information, model node information, and model sharing information. The specific data format is as follows:

[0060]

[0061] The data structure of the model node information includes node position, node size, node attribute, node type, node status, node connection information, node number, node name, node comment, data information, etc. The following is an example of a resource node data structure for data input.

[0062]

[0063]

[0064]

[0065] The following is an example of a model operator node configuration data structure:

[0066]

[0067]

[0068] Step S3: Figure 2 As shown, in response to the user's data model parsing request, the backend system parses the data model of the directed acyclic graph DAG, converts the parsed data model into a model operator, and connects the model operator to the data warehouse;

[0069] It should be noted that in step S3, in response to the user's data model parsing request, the backend system parses the data model of the directed acyclic graph DAG, including:

[0070] Use Spark-core and SparkSQL to develop operator program templates;

[0071] Encapsulate the operator program template into a running code package Runtime.jar;

[0072] Configure the running code package Runtime.jar in the big data cluster Spark.

[0073] Connect to the big data cluster Spark through the Livy interface and call the running code package Runtime.jar.

[0074] The step S3 converts the parsed data model into a model operator, and connects the model operator to the data warehouse, including:

[0075] (1) The background processing device calls the livy interface and passes the model's running configuration parameters to the model running code package Runtime.jar. The model running code package Runtime.jar parses the data model parameters of the directed acyclic graph DAG, obtains the input data warehouse information, model operators, output data warehouse information, etc., and puts the response information in different lists.

[0076] (2) After the Spark task starts executing, it first starts to process the input data warehouse information, model operators, and output data warehouse information. According to the predefined data warehouse type, it loads the spark connector of the corresponding data warehouse and loads the data into the DataFrame to prepare for the subsequent model operator operation. It supports data warehouses of types such as Hive, Guassdb, Gbase, TiDB, and Clickhouse.

[0077] (3) Then execute the model operator, convert the model operator information into an ordered set of model operators, loop through the ordered set of model operators, and according to the model operator type, the operator factory will create an operator instance based on the operator name reflection, and load the required operators through reflection, such as intersection, union, difference, grouping, filtering, etc. At the same time, connect to the actual data warehouse, execute the operators in sequence, and process the data according to the arranged process. The data input and output between operators are connected in series through DataFrame;.

[0078] (4) Finally, when executing data output, a table structure will be created in the corresponding data warehouse based on the output parameters. After the workflow is completed, the result data will be output to the result table.

[0079] Step S4: In response to the user's computing request, the model operator is submitted to the big data cluster through the REST API for data analysis and mining.

[0080] Compared with the prior art, the visual modeling method based on big data provided by the embodiment of the present invention has the following beneficial effects:

[0081] 1. The present invention uses a visual drag-and-drop layout to drag various data resources, components and operators, arrange data resources and model operators, and implement filtering, grouping, intersection, union, difference, deduplication, cleaning and output processing of data resources, generate a data model of a directed acyclic graph DAG in a specific format, and precipitate previous offline and paper-based business knowledge and experience into an intuitive and clear digital model.

[0082] 2. The back-end system parses the data model, converts it into model operators and connects it to the data warehouse. It submits it to the big data cluster through the REST API for data analysis and mining, realizes batch offline computing, streaming computing, and batch-stream integrated computing, and creates a simple and easy-to-use big data mining tool for non-technical personnel.

[0083] 3. The present invention supports hybrid computing of multiple data sources and multiple data warehouses.

[0084] like Figure 3 As shown, an embodiment of the present invention further provides a visual modeling device based on big data, the device comprising:

[0085] The data resource module 1 is used to provide data resources in response to a user's request for building a data model, and the data resources include a data source and a data warehouse.

[0086] It should be noted that the data resource module 1 includes a data source 11 and a data warehouse 12. The data source 11 supports docking with relational databases, message middleware, FTP resources, etc., such as Mysql, Kafka, Oracle, Sql server, Postgresql, DM and more than 30 other databases, and synchronizes the above-mentioned types of data to the data warehouse 12; the data warehouse 12 supports common databases such as Hive, Guassdb, Gbase, TiDB, Clickhouse, etc.

[0087] The data processing module 2 is used to perform a first preprocessing on the data resources in response to the user's data model building request.

[0088] It should be noted that the data processing module 2 includes data source management 21, data access 22, data cleaning 23 and data scheduling 24. The data source management 21 is configured to connect various data sources 11. In order to realize the plug-in access of the data source, the data processing module 2 is implemented using JAVA SPI and JAVA custom class loader classloader to realize data access and table field information viewing. The specific implementation method is to define a top-level general interface, such as an interface interface. The general interface includes methods such as obtaining data source connection, executing queries, obtaining data tables, and returning table field information. The corresponding data source implements the general interface method to form a plug-in package. According to the JAVA SPI requirements, a configuration file with the file name of the full path of the interface is placed in the META-INF\services of the JAR package, and the file content is the full path of the implementation class. At the same time, a custom classloader loader is developed to specify the location of the plug-in package and dynamically load the plug-in package to effectively avoid conflicts caused by the introduction of multiple plug-ins in the project.

[0089] The data processing module 2 synchronizes the data from the data source 11 in batches to the Hive in the data warehouse 12 or other data warehouses supported by this system through data access, data cleaning and data scheduling. The data is cleaned and processed during the data synchronization process, and the data is managed in layers. The specific implementation method is to use the open source datax data integration tool, and through secondary development of it, a more complex data cleaning and conversion function is realized. The data scheduling tool uses XXL-JOB to realize the timing, cycle and other scheduling methods of data. After the data processing module 2 synchronizes the data to the data warehouse 12, the data administrator can configure and authorize the data resources. The authorization methods include three types: user, role, organization, etc., to meet the hierarchical and classified use of data resources.

[0090] A visual modeling module 3, configured to perform a second preprocessing on the data resources in response to a data model building request from a user, so as to form a data model of a directed acyclic graph DAG;

[0091] It should be noted that the visual modeling module 3 is developed using JavaScript, Html5, Vue, and NodeJS, and communicates with the background processing module 4 through HTTP REST and WebSocket to achieve functional operations such as data loading, data preview, operator parameter configuration, and model saving.

[0092] The visual modeling module 3 is used to respond to the user's data model building request, drag and drop the model operator and the first pre-processed data resources on the visual modeling canvas, and save the DAG information of the data modeling, including data resources, model operators, operating parameters, etc. The data format of the model includes model number, model name, model description, model group number, creator, creation time, model parameter information, model label information, model node information, and model sharing information. The specific data format is as follows:

[0093]

[0094] The data structure of the model node information includes node position, node size, node attribute, node type, node status, node connection information, node number, node name, node comment, data information, etc. The following is an example of a resource node data structure for data input.

[0095]

[0096]

[0097]

[0098] The following is an example of a model operator node configuration data structure:

[0099]

[0100]

[0101] Drag the data resources and model operators required for the data model onto the canvas. Through the arrangement and configuration of data resources and operators, filter, group, intersect, union, difference, remove duplicates, cleanse and output data resources to form a directed acyclic graph (DAG) data model in a specific format, trigger the save operation function, and save the directed acyclic DAG data model to the business database.

[0102] The background processing module 4 is used to respond to the user's data model parsing request. The backend system parses the data model of the directed acyclic graph DAG, converts the parsed data model into a model operator, and connects the model operator to the data warehouse.

[0103] It should be noted that if Figure 3 As shown, the background processing module 4 parses the configuration file of the data model of the directed acyclic graph DAG to meet the two situations of local operation preview and cluster operation. The local operation preview is that the user calls the sampling data in real time during the model construction process to realize the real-time viewing of the model operation results during the model construction process. Cluster operation is that after the model is built, the model configuration is parsed and submitted to the big data computing module 5, and data mining and analysis are performed through the big data cluster computing capability. In the process of parsing the model configuration file DAG, the data warehouse type is parsed, and the sparkconnector technology is adopted to realize the simultaneous connection of a single model to multiple data warehouses, thereby realizing the hybrid calculation of multi-source data.

[0104] Specifically, use Spark-core and SparkSQL to develop operator running templates, encapsulate the operator running templates into an executable code package Runtime.jar, configure the code package Runtime.jar in the Spark big data cluster, connect to the big data cluster Spark through Livy and call the code package Runtime.jar.

[0105] The background processing module 4 calls the interface of livy and passes the model's running configuration parameters to the model running code package Runtime.jar. The model running code package Runtime.jar parses the model parameters, obtains the input data warehouse information, model operators, output data warehouse information, etc., and puts the response information in different lists.

[0106] After the Spark task starts executing, it first starts to process the input data warehouse information, model operators, and output data warehouse information. According to the predefined data warehouse type, it loads the spark connector of the corresponding data warehouse and loads the data into the DataFrame to prepare for the subsequent model operator operation. It supports data warehouses of types such as Hive, Guassdb, Gbase, TiDB, and Clickhouse.

[0107] The big data computing module 5 is used to respond to the user's computing request and submit the model operator to the big data cluster for data analysis and mining.

[0108] It should be noted that if Figure 3As shown in the figure, execute the model operator, convert the model operator information into an ordered set of model operators, loop the ordered set of model operators, and according to the model operator type, the operator factory will reflect and create operator instances based on the operator name, such as intersection, union, difference, grouping, filtering, etc. Operators are executed in sequence, and the data input and data output between operators are connected in series through DataFrame. Finally, data output is executed, and the table structure is created in the corresponding data warehouse according to the output parameters. After the workflow is completed, the result data is output to the result table.

[0109] Compared with the prior art, the beneficial effects of a visual modeling device based on big data provided by an embodiment of the present invention are the same as the beneficial effects of a visual modeling method based on big data described in the above technical solution, which will not be elaborated here.

[0110] In addition, an embodiment of the present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are respectively connected via a bus. When the computer program is executed by the processor, each process of the above-mentioned embodiment of a visual modeling method based on big data is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0111] In addition, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes of an embodiment of a visual modeling method based on big data are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0112] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technical solution that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A visualization modeling method based on big data, characterized in that: include: Step S1: in response to a user's data model building request, performing a first preprocessing on data resources, the data resources including a data source and a data warehouse; In the step S1, the data resources are first preprocessed, including: data access, data cleaning and data scheduling are performed on the data in multiple data sources, and then the data in the data sources enter the data warehouse; Step S2: in response to the user's data model building request, performing a second preprocessing on the data resources to form a data model of a directed acyclic graph DAG; The second preprocessing of the data resources in step S2 includes: Drag the model operator and the first preprocessed data resource onto the visual modeling canvas, form a directed acyclic graph DAG data model by arranging and configuring the data resources and the model operator, define the data protocol, trigger the save operation function, and save the directed acyclic graph DAG data model to the business database; in the step S2, it also includes: responding to the user's data model construction request, during the data model construction process, by real-time calling the sampled data to preview the operation result of the data model in real time; Step S3: In response to the user's data model parsing request, the backend system parses the data model of the directed acyclic graph DAG, converts the parsed data model into a model operator, and connects the model operator to the data warehouse; In step S3, in response to the user's data model parsing request, the backend system parses the data model of the directed acyclic graph DAG, including: Use Spark-core and SparkSQL to develop operator program templates; Encapsulate the operator program template into a running code package Runtime.jar; Configure the running code package Runtime.jar in the big data cluster Spark. Connect to the big data cluster Spark through the Livy interface and call the running code package Runtime.jar; Step S4: In response to the user's computing request, submit the model operator to the big data cluster for data analysis and mining; The step S3 converts the parsed data model into a model operator, and connects the model operator to the data warehouse, including: Call the livy interface to pass the model's running configuration parameters to the model running code package Runtime.jar. The model running code package Runtime.jar parses the data model parameters of the directed acyclic graph DAG, obtains the input data warehouse information, model operators, and output data warehouse information, and puts the response information in different lists. After the Spark task starts executing, it processes the input data warehouse information, model operators, and output data warehouse information. According to the predefined data warehouse type, it loads the spark connector of the corresponding data warehouse and loads the data into the DataFrame to prepare for the subsequent model operator operation. Execute the model operator, convert the model operator information into an ordered set of model operators, loop through the ordered set of model operators, and according to the model operator type, the operator factory creates an operator instance based on the operator name reflection, loads the required operator through reflection, and connects to the actual data warehouse at the same time, executes the operators in sequence to process the data according to the arranged process, and the data input and output between operators are connected in series through DataFrame; Execute data output and create a table structure in the corresponding data warehouse according to the output parameters. After the workflow is completed, the result data is output to the result table.

2. The visualization modeling method based on big data according to claim 1, characterized in that: The data sources include Mysql database, Kafka database, Oracle database, Sql server database, Postgresql database, DM database, message middleware and FTP resources; the data warehouse includes Hive database, Guassdb database, Gbase database, TiDB database and Clickhouse.

3. A visual modeling device based on big data, characterized in that: include: A data resource module, used to respond to a user's data model building request and provide data resources, wherein the data resources include a data source and a data warehouse; A data processing module, configured to perform a first preprocessing on data resources in response to a user's request for building a data model; Performing a first preprocessing on the data resources, including: performing data access, data cleaning and data scheduling on the data in multiple data sources, and then the data in the data sources enters the data warehouse; A visual modeling module, configured to perform a second preprocessing on the data resources in response to a data model building request from a user, so as to form a data model of a directed acyclic graph DAG; In the visual modeling module, the second preprocessing of data resources includes: Drag the model operator and the first preprocessed data resources onto the visual modeling canvas, form a directed acyclic graph (DAG) data model by arranging and configuring the data resources and model operators, define the data protocol, trigger the save operation function, and save the directed acyclic graph (DAG) data model to the business database; respond to the user's data model building request, and preview the operation results of the data model in real time by calling the sample data in real time during the data model building process; A background processing module, which is used to respond to a user's data model parsing request, and the backend system parses the data model of the directed acyclic graph DAG, converts the parsed data model into a model operator, and connects the model operator to the data warehouse; In the background processing module, in response to the user's data model parsing request, the backend system parses the data model of the directed acyclic graph DAG, including: Use Spark-core and SparkSQL to develop operator program templates; Encapsulate the operator program template into a running code package Runtime.jar; Configure the running code package Runtime.jar in the big data cluster Spark. Connect to the big data cluster Spark through the Livy interface and call the running code package Runtime.jar; A big data computing module, used to respond to a user's computing request and submit the model operator to a big data cluster for data analysis and mining; Converting the parsed data model into a model operator and connecting the model operator to a data warehouse includes: Call the livy interface to pass the model's running configuration parameters to the model running code package Runtime.jar. The model running code package Runtime.jar parses the data model parameters of the directed acyclic graph DAG, obtains the input data warehouse information, model operators, and output data warehouse information, and puts the response information in different lists. After the Spark task starts executing, it processes the input data warehouse information, model operators, and output data warehouse information. According to the predefined data warehouse type, it loads the spark connector of the corresponding data warehouse and loads the data into the DataFrame to prepare for the subsequent model operator operation. Execute the model operator, convert the model operator information into an ordered set of model operators, loop through the ordered set of model operators, and according to the model operator type, the operator factory creates an operator instance based on the operator name reflection, loads the required operator through reflection, and connects to the actual data warehouse at the same time, executes the operators in sequence to process the data according to the arranged process, and the data input and output between operators are connected in series through DataFrame; Execute data output and create a table structure in the corresponding data warehouse according to the output parameters. After the workflow is completed, the result data is output to the result table.

4. An electronic device, comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, wherein: When the computer program is executed by the processor, the steps in the big data-based visual modeling method as described in any one of claims 1-2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the visual modeling method based on big data as described in any one of claims 1-2 are implemented.

Citation Information

Patent Citations

  • Data stream-oriented big data processing method and device and computer equipment

    CN114217785A

  • Visual data modeling system and method

    CN115757603A

  • System and Method for Big Data Stream Modeling

    KR101687239B1

  • Method of generating a logical data model, physical data model, extraction routines and load routines

    US6490590B1