Intelligent calculation system supporting multiple data sources and data analysis method thereof

Abstract

The invention relates to an intelligent computing system which supports multiple data sources and can carry out data real-time streaming computation, and aims to provide a real-time streaming computation capability which supports multiple data sources. The system allows a user to register a new group or locally deploy in an SaaS environment, configures an operator flow through a user-friendly interface, realizes flexible data processing flow custom configuration, provides flexible algorithm configuration capability, and provides dynamic data source configuration, so that the user can configure different data sources according to actual needs, and the user experience is improved. Data are read and stored, safe physical isolation of the data is achieved, meanwhile, the safety of the user data is considered, and the system adopts national secret encryption storage. And a plurality of data source access functions are provided, so that a user can carry out data entry in other forms such as excel files and interfaces. When the operator stream is operated, the system adopts stream-oriented computation, data is read and computed at the same time, blocking is not needed, the operator stream computation efficiency is improved to a great extent, and hardware consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of big data real-time processing, and is a large data volume real-time streaming computing system that supports flexible configuration of multiple data sources. Background Art

[0002] In the field of data processing, especially in big data analysis and real-time data processing, there are problems such as low data processing efficiency when the data volume is large, high hardware configuration, and inflexible configuration of algorithms. In the data processing process, the data processing system is often limited by the diversity of data source types and the real-time nature of data processing, and cannot effectively provide flexible data processing solutions. Summary of the invention

[0003] The present invention aims to provide an intelligent computing system that can support multiple data sources, including but not limited to relational databases, document databases, interfaces, and file types. The system allows users to configure operator streams through a user-friendly interface, and can perform real-time streaming computing of data to achieve real-time processing and storage of data.

[0004] In order to achieve the above object, the present invention adopts the following technical solution.

[0005] An intelligent computing system that supports multiple data sources and can process and analyze data includes the following steps.

[0006] Step 1: First, in the Group Management Office, each tenant enters their own data source configuration for the storage address of the algorithm center and the network disk.

[0007] Step 2: Configure your own operator permissions in the group management office for subsequent operator flow algorithm configuration.

[0008] Step 3: Drag the operator on the operator flow canvas to build your own operator flow calculation process, including selecting the input operator data source, operator flow algorithm configuration, and operator data output. The configuration here will read the configuration information of the group data source management, and the subsequent operators will read or store the corresponding database based on the configured information. After the configuration is completed, save this configuration as an operator flow file, which can be used as your own algorithm file or as a resource authorization for shared use.

[0009] Step 4: After the file configuration is completed, you can click Run Operator Flow in the next step. Here, the operator flow will initialize the configuration required by the calculation engine based on the configuration file. After successful initialization, each operator starts to start. The initial process node input operator will connect to the corresponding group database based on the data source configured by the group, batch read the configured data source data, and the data enters the operator flow. After the subsequent nodes receive the read data, they start the calculation logic for real-time calculation. The calculated data flows to the next operator node, and at the same time receives the data transferred by the previous operator for calculation. The data will continue to flow and calculate in the operator flow until the calculation process ends and is output to the corresponding configuration receiving table.

[0010] Step 5: When no more data is read into the operator stream, each operator will stop automatically if it no longer receives data (if an error occurs during the calculation, the operator will also stop). After the calculation is completed, you can view the operation records and synchronize the operation results with the database. During this period, considering the user data security plan, the national secret algorithm can be configured for encryption of sensitive data, and encrypted data transmission is also encrypted.

[0011] Preferably, in step 1, the group administrator performs data management configuration for different group tenants, thereby achieving physical data isolation for different tenants and ensuring data security and configuration flexibility for different tenants.

[0012] Preferably, in step one, different data are grouped into databases, which also ensures system stability and availability. Different groups will not suffer from reduced database performance due to the common storage of large amounts of data, and groups can configure their own databases according to their own needs. If one group database is unavailable, it will not affect the use of functions of other groups.

[0013] Preferably, in step 3, users can freely assemble their own operator flow calculation process in the operator flow canvas interface according to their needs, including data source specification, data output specification, and algorithm flow configuration. At the same time, the configured algorithm flow can be shared as a resource file. For complex functions, there is no need to repeat the workload, and the operator flows can be combined and run, which increases the configuration flexibility.

[0014] Preferably, in step 4, the algorithm execution engine adopts streaming processing, data flows in from the input operator and immediately flows to the next operator for calculation. After the calculation is completed, the data continues to flow downward and receives the data from the previous operator for calculation. In this process, all operators process data at the same time, which solves the problem in traditional calculations that all data must be calculated before flowing downward, ensuring data processing efficiency and resource utilization.

[0015] Preferably, in step 4, the operator in the operator flow receives data for processing, and is closed if no data is received, which greatly utilizes resources without causing resource waste. When the operator flow is in the error reporting stage, all operator flow operations will be stopped.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows.

[0017] A-The system supports a wide range of data source types, including relational databases, document databases, various interfaces, and multiple file formats, providing users with unprecedented flexibility, enabling them to easily integrate and process data from different sources.

[0018] The B-system provides an intuitive drag-and-drop interface that allows users to configure operator flows without writing complex code, greatly reducing the technical threshold of the data processing process.

[0019] C-Users can choose to store data in the data network disk provided by the system, or configure it as a private library to meet different security and privacy requirements. In addition, the use of the SM4 national secret encryption algorithm further protects the security of database connection information. The configured operator flow can be saved as a file for easy management and reuse, and can be shared with other users or authorized to other users, promoting collaboration and knowledge sharing.

[0020] D-Users can dynamically configure the algorithm center and data network disk so that the system can adapt to changing business needs and data processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0022] Figure 1 An intelligent computing system process and data distribution diagram provided for an embodiment of the present invention.

[0023] Figure 2 A schematic diagram of a multi-data source configuration provided by an embodiment of the present invention.

[0024] Figure 3 A system architecture diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The invention provides a technical solution.

[0027] Please refer to Figure 3 , an intelligent computing system that supports multiple data sources and can process and analyze data includes the following contents.

[0028] Data network disk: user data storage center service, users can configure data connections, data sources, synchronize user data, provide storage capabilities, etc.

[0029] Algorithm Center: Operator flow computing center, users can flexibly configure algorithm operator flows, run operator flows in the algorithm center, and view operator flow logs in real time.

[0030] Public configuration: Group administrator permission management configuration, which can configure group menus, operators, and data source permissions.

[0031] BI visualization: graphical display of calculation results.

[0032] Please refer to Figure 1 The example steps are as follows.

[0033] Step 1: First, in the Group Management Office, each tenant enters their own data source configuration for the storage address of the algorithm center and the network disk.

[0034] Step 2: Configure your own operator permissions in the group management office for subsequent operator flow algorithm configuration.

[0035] Step 3: Drag the operator on the operator flow canvas to build your own operator flow calculation process, including selecting the input operator data source, operator flow algorithm configuration, and operator data output. The configuration here will read the configuration information of the group data source management, and the subsequent operators will read or store the corresponding database based on the configured information. After the configuration is completed, save this configuration as an operator flow file, which can be used as your own algorithm file or as a resource authorization for shared use.

[0036] Step 4: After the file configuration is completed, you can click Run Operator Flow in the next step. Here, the operator flow will initialize the configuration required by the calculation engine based on the configuration file. After successful initialization, each operator starts to start. The initial process node input operator will connect to the corresponding group database based on the data source configured by the group, batch read the configured data source data, and the data enters the operator flow. After the subsequent nodes receive the read data, they start the calculation logic for real-time calculation. The calculated data flows to the next operator node, and at the same time receives the data transferred by the previous operator for calculation. The data will continue to flow and calculate in the operator flow until the calculation process ends and is output to the corresponding configuration receiving table.

[0037] Step 5: When no more data is read into the operator stream, each operator will stop automatically if it no longer receives data (if an error occurs during the calculation, the operator will also stop). After the calculation is completed, you can view the operation records and synchronize the operation results with the database. During this period, considering the user data security plan, the national secret algorithm can be configured for encryption of sensitive data, and encrypted data transmission is also encrypted.

[0038] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent computing system that supports real-time streaming computing from multiple data sources: Step 1: First, in the Group Management Office, each tenant enters their own data source configuration for the storage address of the algorithm center and the network disk; Step 2: Configure your own operator permissions in the group management office for subsequent operator flow algorithm configuration; Step 3: On the operator flow canvas, drag the operator to build your own operator flow calculation process, including selecting the input operator data source, operator flow algorithm configuration, and operator data output. The configuration here will read the configuration information of the group data source management, and the subsequent operators will read or store the corresponding database based on the configured information. After the configuration is completed, save this configuration as an operator flow file. This file can be used as your own algorithm file or as a resource authorization for shared use; Step 4: After the file configuration is completed, you can click Run Operator Flow in the next step. The operator flow here will initialize the configuration required by the calculation engine based on the configuration file. After the initialization is successful, each operator starts to start. The input operator of the initial process node will connect to the corresponding group database based on the data source configured by the group, and batch read the configured data source data. The data enters the operator flow. After the subsequent nodes receive the read data, they start the calculation logic for real-time calculation. The calculated data flows to the next operator node, and at the same time receives the data transferred by the previous operator for calculation. The data will continue to flow and calculate in the operator flow until the calculation process ends and is output to the corresponding configuration receiving table; Step 5: When no more data is read into the operator stream, each operator will stop automatically if it no longer receives data (if an error occurs during the calculation, the operator will also stop). After the calculation is completed, you can view the operation records and synchronize the operation results with the database. During this period, considering the user data security plan, the national secret algorithm can be configured for encryption of sensitive data, and encrypted data transmission is also encrypted.

2. According to claim 1, an intelligent computing system that supports multiple data sources and can perform real-time data streaming computing is characterized in that: In step one, the group management source can implement data source configuration in the group management interface, which ensures the database pressure and security of different tenants at the SaaS level. Customers can provide their own data based on their own needs, and read and store data in their own database to avoid data leakage. At the same time, the configuration of different tenants realizes the group tenant data sub-library from another level, ensuring the high availability of the database, so as not to cause a group data anomaly, resulting in database unavailability, and thus system unavailability. And through the configured group database, the database cutting process is relatively simple, without changing the code, only needing to configure the connection and start.

3. According to claim 1, the intelligent computing system that supports multiple data sources and can perform real-time data streaming computing is characterized in that: In step 2, the administrator can manage the permissions of the operator.

4. According to claim 1, the intelligent computing system that supports multiple data sources and can perform real-time data streaming computing is characterized in that: In step three, enter the canvas of the algorithm center, and configure and run the operator flow based on the group-authorized operator. At this stage, users can configure their own algorithm flow, operator flow data source, and data source required to store the calculation output operator according to their needs. After the configuration here is completed, you can run this operator flow and obtain the operation result data. You can also store this operator flow as a resource file in the network disk, and authorize or share it with people in the same group. This operator flow file can be shared between different groups or systems by exporting and importing.

5. According to claim 1, the intelligent computing system that supports multiple data sources and can perform real-time data streaming computing is characterized in that: In step 4, the operator flow will parse the data configured on the operator flow page before running, first parse it into the configuration data required by the streaming computing engine, and then initialize each process node based on the configuration data. After all the initialization actions are completed, the operator flow starts to start, the input operator starts to connect to the configured data source, and reads the corresponding data from the data source, and flows to the next operator. The operator that receives the data starts to calculate, and the result of the calculation is transferred to the next operator, and then continues to receive the data transferred from the previous operator flow. After all the data in the operator flow is calculated, the operator flow calculation ends. In this process, the operator flow calculation result will flow to the output operator, and the output operator reads the corresponding operator flow connection configuration and outputs the calculated data to the corresponding result table.

6. The intelligent computing system supporting multiple data sources and capable of real-time data streaming computing according to claim 1 is characterized in that: In step five, the operator flow will automatically stop if an error occurs or data processing is completed. After the calculation is completed, the operator flow operation record can be queried at the network disk operation record, and the operator flow data that has been successfully run has been stored in the configured database. The sensitive information in the database is encrypted and transmitted using national secrets to ensure data security.

Images