Low-code platform multi-source heterogeneous data integration system and method

Abstract

The invention discloses a low-code platform multi-source heterogeneous data integration system and method, and the system comprises a dynamic data source management module which comprises the functions of data source configuration management, dynamic data source routing, connection pool management and data source health examination; the unified data access abstraction layer has the functions of JOOQ integration, SQL dialect conversion, data type mapping and transaction management; the unified instruction system comprises an instruction parser, an instruction executor and an instruction packaging and composite operation supporting function; and the visual development module has the functions of table structure design, form component mapping, data operation interface and statistical analysis. A layered architecture design is adopted, and unified integration and management of multi-source heterogeneous data are realized through dynamic data source management, a unified data access abstraction layer, an instruction analysis system and a visual development interface.

Description

Technical Field

[0001] The present invention relates to the technical field of computer software, and in particular to a low-code platform multi-source heterogeneous data integration system and method, which is specifically used in fields such as enterprise-level application development, data integration, and visualization development. Background Art

[0002] Existing enterprise-level application development and data integration have the following problems: 1. Data source heterogeneity Enterprises typically use multiple database systems (MySQL, PostgreSQL, Oracle, etc.). Each database has different SQL dialects, data types, and connection methods, which requires developers to write different codes for different databases, increasing development complexity and maintenance costs.

[0003] 2. Data access layer complexity Traditional data access requires writing specific DAO layer code for each database. When supporting new database types, a lot of code modification and testing work is required.

[0004] 3. Low development efficiency Traditional enterprise-level application development requires writing a large amount of CRUD operation code, which is repetitive, has a long development cycle, and is prone to errors.

[0005] 4. Data integration is difficult Data integration between different data sources usually requires writing complex ETL scripts, which are difficult to maintain and lack a unified data operation interface.

[0006] 5. Limitations of Visual Development Most existing low-code platforms only support a single database, or have functional limitations in supporting multiple databases, and cannot meet the complex data integration needs of enterprises.

[0007] The main difficulties in solving these problems lie in: how to provide a unified data access abstraction layer while ensuring performance, which can not only shield the differences between different databases but also make full use of the unique functions of each database; how to design a flexible instruction system that can express complex data operation requirements in a unified manner. Summary of the Invention

[0008] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0009] In view of the above problems existing in the prior art enterprise-level application development and data integration platform, the present application is proposed.

[0010] Therefore, the technical problem solved by the present application is how to construct a unified multi-source heterogeneous data integration system and method to realize unified access and operation of different types of databases, while providing visual low-code development capabilities to reduce the complexity and cost of enterprise-level application development.

[0011] To solve the above technical problems, the present application provides the following technical solutions: a low-code platform multi-source heterogeneous data integration system, the system comprising the following modules: a dynamic data source management module for supporting dynamic configuration and management of multiple database types, including data source configuration management, dynamic data source routing, connection pool management, and data source health check; a unified data access abstraction layer for realizing unified access to multiple databases based on JOOQ, including SQL dialect automatic identification, data type mapping, and cross-data source transaction management; a unified instruction system for parsing and executing data operation instructions in JSON format, including an instruction parser, an instruction executor, instruction encapsulation, and composite instruction support for complex data operations; a visual development module providing a visual table structure design interface, form component mapping, data operation interface, and statistical analysis functions.

[0012] As a preferred scheme of the low-code platform multi-source heterogeneous data integration system of the present application, the dynamic data source management module comprises: dynamic data source routing based on DynamicRoutingDataSource to realize dynamic switching of data sources; connection pool management using Hikari connection pool technology to realize efficient database connection management; and data source health check providing data source connection testing and state monitoring functions.

[0013] As a preferred scheme of the low-code platform multi-source heterogeneous data integration system of the present application, the unified data access abstraction layer comprises: JOOQ integration to realize unified access to multiple databases through JOOQ, automatically identify database dialects, and generate corresponding SQL statements; data type mapping to provide a unified data type mapping mechanism to shield the data type differences of different databases; and transaction management to support distributed transaction processing across data sources.

[0014] As a preferred solution of the low-code platform multi-source heterogeneous data integration system described in the present invention, the unified instruction system includes: an instruction parser for parsing data operation instructions in JSON format; an instruction executor for uniformly executing various database operation instructions, including data addition, deletion, modification and query operations; instruction encapsulation, using the DbCommand object as a unified data operation instruction carrier; and compound operation support, supporting complex multi-table joint query and batch operation compound instructions.

[0015] As a preferred solution of the low-code platform multi-source heterogeneous data integration system described in the present invention, the visual development module includes: a table structure design interface that supports the configuration of field types and constraints; a form component mapping that maps front-end form components to database fields and supports multiple component types; a data operation interface that provides a unified data addition, deletion, modification and query operation interface; a statistical analysis function with a built-in statistical analysis query builder that supports complex data analysis needs.

[0016] As a preferred solution of the low-code platform multi-source heterogeneous data integration system described in the present invention, the system also supports the following functions: dynamic data source routing, dynamically switching to the corresponding data source according to the data source information in DbCommand; data operation execution, parsing and executing JSON format instructions through CommandExecutor, supporting data addition, deletion, modification and query operations; data integration capabilities, supporting multi-table joint query and complex data integration operations across data sources; data security, built-in data encryption and permission management functions to ensure data security.

[0017] To solve the above technical problems, the present invention also provides the following technical solutions: a method for realizing multi-source heterogeneous data integration on a low-code platform, adopting the multi-source heterogeneous data integration system of the low-code platform described above, characterized in that the method includes the following steps: data source configuration and management: initializing the dynamic data source manager through the DataSourceConfig configuration class, supporting dynamic configuration of multiple database types; unified instruction parsing and execution: parsing instructions in JSON format through CommandParser, and performing data operations through CommandExecutor; dynamic data source routing: dynamically switching to the corresponding data source according to the data source information in DbCommand; JOOQ unified data access: realizing unified access to multiple databases through JOOQ, automatically identifying database dialects and generating corresponding SQL statements; data operation execution: executing corresponding database operations according to the operation type, including data addition, deletion, modification and query operations.

[0018] As a preferred solution of the method for realizing multi-source heterogeneous data integration of a low-code platform described in the present invention, the data source configuration and management steps include: initializing the dynamic data source manager; selecting a suitable data source creator according to the database driver type; and supporting the configuration of default data sources and extended data sources.

[0019] As a preferred solution of the method for realizing multi-source heterogeneous data integration on a low-code platform described in the present invention, the unified instruction parsing and execution steps include: parsing instructions in JSON format into DbCommand objects; executing data operation instructions, supporting data addition, deletion, modification and query operations; supporting compound instructions, including multi-table joint query and batch operations.

[0020] As a preferred solution of the method for realizing multi-source heterogeneous data integration on a low-code platform described in the present invention, the JOOQ unified data access step includes: obtaining database metadata, identifying the database type and version; selecting the corresponding SQL dialect according to the database type; generating and executing the corresponding SQL statement through JOOQ.

[0021] The present invention provides a low-code platform multi-source heterogeneous data integration system and method, which has the following beneficial effects: 1. Unity Through a unified data access abstraction layer, unified operations on multiple heterogeneous databases are achieved, and developers do not need to worry about the specific types and differences of the underlying databases.

[0022] 2. Efficiency The use of dynamic data source routing and connection pool management technology improves database access efficiency and reduces connection overhead.

[0023] 3. Flexibility The unified instruction system supports complex data operation requirements and can implement various database operations through JSON configuration without writing code.

[0024] 4. Scalability The modular design makes the system easy to expand and supports the rapid integration of new database types.

[0025] 5. Development Efficiency The visual development interface significantly lowers the technical threshold for application development and improves development efficiency.

[0026] 6. Cost savings Reduced duplication of development work and lowered system maintenance costs.

[0027] 7. Data Security Built-in data encryption and permission management functions ensure data security.

[0028] Solve how to build a unified multi-source heterogeneous data integration system, realize unified access and operation to different types of databases, and provide visual low-code development capability, reduce the complexity and cost of enterprise application development. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them: Figure 1 The system overall architecture diagram of the low-code platform multi-source heterogeneous data integration system provided by the present application shows the hierarchical relationship and data flow direction between each module.

[0030] Figure 2 The data operation flowchart of the low-code platform multi-source heterogeneous data integration system provided by the present application shows the complete time sequence from user request to database operation.

[0031] Figure 3 The multi-data source integration processing flowchart of the low-code platform multi-source heterogeneous data integration system provided by the present application describes in detail the logical flow of the system processing multiple databases.

[0032] Figure 4 The system core component relationship diagram of the low-code platform multi-source heterogeneous data integration system provided by the present application shows the relationship between the main classes and interfaces. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0034] In view of the existing problems of enterprise-level application development and data integration, such as data source heterogeneity, data access layer complexity, low development efficiency, data integration difficulty and visual development limitation.

[0035] Reference Figure 1 Figure 4 ​The application provides a low-code platform multi-source heterogeneous data integration system, which adopts a hierarchical architecture design, realizes unified integration and management of multi-source heterogeneous data through dynamic data source management, unified data access abstraction layer, instruction analysis system and visual development interface.

[0036] Specifically includes the following components: The dynamic data source management module is used for supporting dynamic configuration and management of multiple database types, including data source configuration management, dynamic data source routing, connection pool management and data source health check; The unified data access abstraction layer is based on JOOQ to realize unified access to multiple databases, including SQL dialect automatic identification, data type mapping and cross-data source transaction management; The unified instruction system is used for parsing and executing data operation instructions in JSON format, including instruction parser, instruction executor, instruction encapsulation and composite instruction support for complex data operations; The visual development module provides visual table structure design interface, form component mapping, data operation interface and statistical analysis function.

[0037] It should be noted that the system is analyzed according to each module, and the summary is as follows: 1. Dynamic data source management module Dynamic data source routing: based on DynamicRoutingDataSource to realize dynamic switching of data source. When the system receives a data operation request, it can flexibly switch to the corresponding data source according to the data source information in the request to process data of different data sources.

[0038] Connection pool management: Hikari connection pool technology is used to realize efficient database connection management. Hikari connection pool can optimize the creation, use and release process of database connection, reduce connection overhead and improve database access efficiency.

[0039] Data source health check: provides data source connection test and state monitoring function. The connection state of the data source is checked regularly to ensure that the data source is in a usable state, and connection exceptions are found and handled in time.

[0040] Data source configuration management: the dynamic data source manager is initialized through the DataSourceConfig configuration class to support dynamic configuration of multiple database types. According to the database driver type, select the appropriate data source creator, and support the configuration of default data source and extended data source.

[0041] 2. Unified data access abstraction layer JOOQ integration: Unified access to multiple databases is achieved through JOOQ, which automatically identifies database dialects and generates corresponding SQL statements. The specific steps are to obtain database metadata, identify database type and version; select the corresponding SQL dialect according to the database type; generate and execute the corresponding SQL statement through JOOQ.

[0042] Data type mapping: Provides a unified data type mapping mechanism to shield the differences in data types of different databases. Developers do not need to worry about the differences in data types between different databases when performing data operations.

[0043] Transaction management: Supports distributed transaction processing across data sources. When operations involving multiple data sources are involved, it can ensure data consistency and integrity.

[0044] 3. Unified instruction system Instruction parser: used to parse JSON format data operation instructions, and parse JSON format instructions into DbCommand objects.

[0045] Instruction executor: unified execution of various database operation instructions, including data insertion, deletion, modification and query operations. Through CommandExecutor, JSON format instructions are parsed and executed to support data insertion, deletion, modification and query operations, as well as composite instructions such as multi-table query and batch operation.

[0046] Instruction encapsulation: DbCommand object is used as a unified data operation instruction carrier to facilitate the management and transmission of data operation instructions.

[0047] Composite operation support: Supports complex multi-table query, batch operation and other composite instructions to meet more complex data operation requirements.

[0048] 4. Visual development module Table structure design interface: supports field type and constraint configuration, making it easy for developers to design database table structure.

[0049] Form component mapping: maps front-end form components to database fields, supports multiple component types, and realizes the interaction between front-end forms and databases.

[0050] Data operation interface: provides a unified data insertion, deletion, modification and query operation interface to facilitate user data operation.

[0051] Statistical analysis function: built-in statistical analysis query builder supports complex data analysis requirements to help users analyze data in depth.

[0052] Further, in the dynamic data source management module: Dynamic data source routing, based on DynamicRoutingDataSource to achieve dynamic switching of data sources; Connection pool management, using Hikari connection pool technology to achieve efficient database connection management; Data source health check, providing data source connection testing and status monitoring functions.

[0053] It should be noted that: ①Dynamic data source routing Implementation: Dynamically switch data sources based on the DynamicRoutingDataSource. In the DataSourceConfig class, the dynamicRoutingDataSource method creates a DynamicRoutingDataSource instance and adds a default data source. When the system receives a data operation request, the DynamicRoutingDataSource dynamically switches to the corresponding data source based on the data source information in the request.

[0054] The logical processing is as follows: Configuration class definition: Use the @Configuration annotation to mark the DataSourceConfig class as a Spring configuration class. Spring scans the class at startup and processes it as configuration information.

[0055] Bean definition: Use the @Bean annotation to register the DynamicRoutingDataSource object returned by the dynamicRoutingDataSource method as a Spring Bean, so that the object can be obtained by the Bean name in the Spring application context.

[0056] Dynamic routing data source creation: In the dynamicRoutingDataSource method, create a DynamicRoutingDataSource instance and pass in providers as parameters. providers is an object used to provide data source information.

[0057] Add a default data source: Call the ds.addDataSource method to add the default data source to the DynamicRoutingDataSource instance. The name of the default data source is DEFAULT_DATASOURCE_NAME, and the data source is provided by the primaryDataSource method.

[0058] Return instance: Finally, return the created DynamicRoutingDataSource instance.

[0059] Key technical points: Dynamic data source routing: DynamicRoutingDataSource is used to achieve dynamic switching of data sources. When the system receives a data operation request, DynamicRoutingDataSource will dynamically switch to the corresponding data source for operation based on the data source information in the request.

[0060] Spring configuration: The DataSourceConfig class and dynamicRoutingDataSource method are integrated into the Spring framework using Spring's @Configuration and @Bean annotations, realizing Spring's management of data source configuration.

[0061] Default data source configuration: The ds.addDataSource method is used to add a default data source, ensuring that the system can use the default data source for data operations when no data source information is specified.

[0062] ②Connection pool management Implementation: Hikari connection pool technology is used to achieve efficient database connection management. In the createDataSource method, a data source is created using hikariDataSourceCreator.createDataSource(property). Hikari connection pool can optimize the creation, use, and release process of database connections, reducing connection overhead and improving database access efficiency.

[0063] Key technical points: Database type judgment: The EnhancedDataBaseDialect.isRelationalByDriver method is used to determine the database type, thereby deciding which method to use to create a data source.

[0064] Connection pool management: For relational databases, Hikari connection pool is used to create a data source. Hikari connection pool can optimize the creation, use, and release process of database connections, reducing connection overhead and improving database access efficiency.

[0065] Extensibility: The code reserves interfaces for extending other database types, making it easy to add support for non-relational databases or other special databases in the future.

[0066] Step analysis Step 1: Input parameter check This step ensures that the parameter received by the method is a valid DataSourceProperty object.

[0067] Step 2: Determine the database type Call the EnhancedDataBaseDialect.isRelationalByDriver method to determine whether the database is a relational database based on the driverClassName variable. Ensure that the driverClassName variable is correctly defined and assigned a value.

[0068] Step 3: Create a data source If EnhancedDataBaseDialect.isRelationalByDriver(driverClassName) returns true, it means it is a relational database. In this case, call the hikariDataSourceCreator.createDataSource(property) method to create a data source with the help of the Hikari connection pool and return the data source.

[0069] If EnhancedDataBaseDialect.isRelationalByDriver(driverClassName) returns false, the code does not provide specific processing logic. However, interfaces for extending other database types are provided, where you can add creation logic for other database types.

[0070] ③Data source health check Implementation method: Provide data source connection testing and status monitoring functions, check the connection status of the data source through scheduled tasks or other mechanisms in the system to ensure that the data source is in an available state.

[0071] The logical processing and technical features are described as follows: 1. Dynamic routing data source configuration Technical features: Use Spring's @Configuration and @Bean annotations to mark the DataSourceConfig class as a configuration class, and register the DynamicRoutingDataSource object returned by the dynamicRoutingDataSource method as a Spring Bean.

[0072] Logical processing: First, create a DynamicRoutingDataSource instance and pass in the providers object, which is used to provide data source information.

[0073] Call the addDataSource method to add the default data source to the dynamic routing data source. The default data source is created by the primaryDataSource method.

[0074] Finally, the DynamicRoutingDataSource instance is returned.

[0075] 2. Automatic identification of multiple database types Technical Features: Use the EnhancedDataBaseDialect.isRelationalByDriver method to determine whether the database driver type is a relational database, and select the appropriate data source creator based on the judgment result.

[0076] Logical processing: In the createDataSource method, receive the DataSourceProperty object as a parameter.

[0077] Call the EnhancedDataBaseDialect.isRelationalByDriver method and pass in the driverClassName for judgment.

[0078] If it is a relational database, use the hikariDataSourceCreator.createDataSource method to create a data source.

[0079] An interface for extending other database types is reserved, and processing logic for other database types can be added in the else branch.

[0080] 3. Primary data source creation Technical features: In the primaryDataSource method, create a DataSourceProperty object, set the data source properties, and select the appropriate data source creator based on the driver type.

[0081] Logical processing: Create a DataSourceProperty object.

[0082] Set data source properties, such as password, etc.

[0083] Call the EnhancedDataBaseDialect.isRelationalByDriver method, pass in driverClassName to determine.

[0084] If it is a relational database, use the hikariDataSourceCreator.createDataSource method to create a data source.

[0085] If it is not a relational database, throw a RuntimeException exception to indicate that the database type is not supported.

[0086] Further, the unified data access abstraction layer includes: JOOQ integration, which provides unified access to multiple databases through JOOQ, automatically identifies database dialects and generates corresponding SQL statements; Data type mapping, which provides a unified data type mapping mechanism to shield the differences in data types between different databases; Transaction management, which supports distributed transaction processing across data sources.

[0087] It should be noted that: ①JOOQ integration Implementation: Through JOOQ, unified access to multiple databases is achieved, and the corresponding SQL statements are automatically generated based on the database dialect. In the getDslContext method, first, the database metadata is obtained, and the corresponding SQL dialect is determined based on the database product name and version number. Then, the Configuration object is created and the connection, dialect, and log listener are set. Finally, the DSL.using(configuration) is used to create the DSLContext object for executing SQL operations.

[0088] The logical processing is described as follows: Get database metadata: Get the metadata information of the database from the input database connection object through the connection.getMetaData() method. The metadata contains various properties of the database, such as the database product name, version number, etc.

[0089] Determine the database type: Extract the database product name from the metadata and use it as the database type for subsequent determination of the corresponding SQL dialect.

[0090] Determine SQL dialect: Call the getSQLDialect method to determine the SQL dialect used by the database based on the database type and version number. Different databases may have different SQL syntax and features, and determining the dialect allows JOOQ to generate SQL statements suitable for the database.

[0091] Create a configuration object: Create a DefaultConfiguration object that configures various parameters for JOOQ to interact with the database.

[0092] Configure connection, dialect, and log listener: Set the database connection, SQL dialect, and log listener provider to the configuration object in sequence. The log listener can record the SQL statements executed by JOOQ, facilitating debugging and monitoring.

[0093] Create and return a DSLContext object: Use the configuration object to call the DSL.using(configuration) method to create a DSLContext object and return it. Developers can use the DSLContext object to perform SQL operations on different databases in a unified manner.

[0094] Technical features Automatic recognition of database dialect: By obtaining database metadata and product names, the type and version of the database are automatically identified, and the corresponding SQL dialect is determined, achieving unified access to multiple databases.

[0095] Use of configuration object: Use the DefaultConfiguration object to centrally manage various configurations for JOOQ to interact with the database, including connection, dialect, and log listener, etc., improving code maintainability and flexibility.

[0096] Logging: By setting DefaultExecuteListenerProvider and JooqSqlLogger, the SQL statements executed by JOOQ can be recorded, facilitating developers in debugging and monitoring.

[0097] Unified SQL operation interface: The returned DSLContext object provides a unified interface for developers to perform SQL operations on different databases, masking the differences between different databases.

[0098] This method receives a database connection object and returns a DSLContext object, which developers can use to perform SQL operations on different databases in a unified manner.

[0099] ②Data type mapping Implementation method: Provide a unified data type mapping mechanism to shield the data type differences between different databases. During the JOOQ integration process, JOOQ will automatically handle the mapping of different database data types.

[0100] ③Transaction management Implementation method: Support distributed transaction processing across data sources, and implement transaction control across data sources through Spring's transaction management mechanism combined with related transaction managers.

[0101] The logical processing is described as follows: Get database metadata: Use the connection.getMetaData() method to get the database metadata from the passed database connection object. The metadata contains various information about the database, such as the database name and version.

[0102] Determine the database type: Extract the database product name from the metadata as the database type, which is used to subsequently determine the corresponding SQL dialect.

[0103] Determine the SQL dialect: Call the getSQLDialect method to get the SQL dialect appropriate for the database based on the database type and version number. Different databases may have different SQL syntax. By determining the dialect, JOOQ can generate SQL statements that conform to the database.

[0104] Create a configuration object: Create a DefaultConfiguration object, which is used to configure the relevant parameters for JOOQ to interact with the database.

[0105] Configure the connection, dialect, and log listener: Set the database connection, SQL dialect, and log listener provider to the configuration object. The log listener can record the SQL statements executed by JOOQ, which is convenient for debugging and monitoring.

[0106] Create and return a DSLContext object: Using the configured DefaultConfiguration object, create a DSLContext object through the DSL.using(configuration) method and return it. Developers can use this object to execute SQL operations on different databases in a unified way.

[0107] Technical features Database dialect automatic recognition: By obtaining database metadata, the database type and version are automatically recognized, and the corresponding SQL dialect is determined, realizing unified access to multiple databases.

[0108] Centralized management of configuration objects: The DefaultConfiguration object is used to centrally manage the configuration of JOOQ interacting with the database, including connection, dialect, and log listener, etc., improving the maintainability and flexibility of the code.

[0109] Logging function: By setting DefaultExecuteListenerProvider and JooqSqlLogger, the executed SQL statements of JOOQ can be recorded, which is convenient for developers to debug and monitor database operations.

[0110] Unified data access interface: The returned DSLContext object provides a unified interface for developers, shielding the differences between different databases, so that developers can perform SQL operations on different databases in the same way.

[0111] Further, the unified instruction system includes: Instruction parser, used to parse JSON format data operation instructions; Instruction executor, unified execution of various database operation instructions, including data insertion, deletion, modification and query operations; Instruction encapsulation, using DbCommand object as a unified data operation instruction carrier; Composite operation support, supporting complex multi-table query, batch operation and other composite instructions.

[0112] It should be noted that: ① Instruction parser Implementation: used to parse JSON format data operation instructions, there are special classes or methods in the system to parse JSON format instructions into DbCommand objects.

[0113] ② Instruction executor Implementation: in the executeDbCommand method of the CommandExecutor class, according to the operation type of the DbCommand object, different methods are called to execute corresponding database operations, including data insertion, deletion, modification and query.

[0114] The logical processing is described as follows: Receive instructions: the executeDbCommand method of the CommandExecutor class receives a DbCommand object as input, which encapsulates specific data operation instructions.

[0115] Determine operation type: Get the operation type through dbCommand.getOperationType(), and use a switch statement to branch according to different operation types.

[0116] Execute corresponding operation: When the operation type is METHOD_GET, call the executeGetCommand method to execute the query operation and return the operation result.

[0117] When the operation type is METHOD_ADD, call the executeAddCommand method to execute the add operation and return the operation result.

[0118] When the operation type is METHOD_UPDATE, call the executeUpdateCommand method to execute the update operation and return the operation result.

[0119] When the operation type is METHOD_REMOVE, call the executeRemoveCommand method to execute the delete operation and return the operation result.

[0120] Technical features Unified instruction execution: The CommandExecutor class provides a unified interface executeDbCommand to execute different types of data operation instructions, which centrally manages the execution logic of different operations and improves the maintainability of the code.

[0121] Operation type distinction: Different operations are handled differently through a switch statement, making the operation logic separate, which is convenient for extension and modification. For example, if you need to add a new operation type, you only need to add a new case branch in the switch statement.

[0122] Encapsulation: The specific operation logic is encapsulated in methods such as executeGetCommand, executeAddCommand, executeUpdateCommand, and executeRemoveCommand, making the code of the executeDbCommand method more concise and improving the reusability of the code.

[0123] This method receives a DbCommand object, executes the corresponding command according to its operation type, and returns the operation result.

[0124] ③ Instruction encapsulation Implementation: By DbCommand object as a unified data operation instruction carrier, the related information of data operation is encapsulated in the DbCommand object, which is convenient for transmission and processing in the system.

[0125] (4) Composite operation support Implementation: Support complex multi-table inquiry, batch operation and other composite instructions. In the executeDbCommand method, different operations can be combined to realize the execution of composite instructions.

[0126] Specifically, the application synchronously provides the following specific code implementation: The logical processing is described as follows: Receive instruction: The executeDbCommand method in the CommandExecutor class receives a DbCommand object, which contains specific data operation instructions.

[0127] Operation type judgment: The operation type is obtained by calling dbCommand.getOperationType(), and then the operation type is judged by using the switch statement.

[0128] Execute corresponding operation: If the operation type is METHOD_GET, the executeGetCommand method is called to execute the query operation, and the operation result is returned.

[0129] If the operation type is METHOD_ADD, the executeAddCommand method is called to execute the addition operation, and the operation result is returned.

[0130] If the operation type is METHOD_UPDATE, the executeUpdateCommand method is called to execute the update operation, and the operation result is returned.

[0131] If the operation type is METHOD_REMOVE, the executeRemoveCommand method is called to execute the deletion operation, and the operation result is returned.

[0132] Technical feature points Unified instruction execution: The CommandExecutor class provides a unified entrance executeDbCommand to process different types of database operation instructions, which centrally manages the parsing and execution logic of the instructions, improves the maintainability and scalability of the code.

[0133] Operation type separation: Use a switch statement to call different processing methods based on different operation types, making the logic of different operations independent of each other and facilitating code modification and expansion. If you need to add a new operation type, simply add a new case branch to the switch statement.

[0134] Encapsulation: Encapsulating the specific operation logic in methods such as executeGetCommand, executeAddCommand, executeUpdateCommand, and executeRemoveCommand makes the code of the executeDbCommand method concise and clear, and also improves the reusability of the code.

[0135] Furthermore, the visual development module includes: Table structure design interface, supporting configuration of field types and constraints; Form component mapping, mapping front-end form components to database fields, supporting multiple component types; Data operation interface, providing a unified data addition, deletion, modification and query operation interface; Statistical analysis function, built-in statistical analysis query builder, supports complex data analysis needs.

[0136] It should be noted that: ①Table structure design interface Implementation method: Supports the configuration of field types and constraints, and provides corresponding forms and controls in the visual interface, allowing users to configure the field types and constraints of the table structure.

[0137] ②Form component mapping Implementation: In the convertFormItemToSchemaItem method, convert the front-end form component information into a database table structure object. Based on the form component type, obtain the corresponding FormItemType from FORM_ITEM_TYPES_MAP and then construct a SchemaObject object.

[0138] The logical processing is described as follows: Get component type: Extract the value of the "component" field from the passed formItem (a JSONObject object representing a form component) and store it in the componentType variable. This value represents the type of the form component.

[0139] Find the form item type: Use componentType as the key to find the corresponding FormItemType object from the FORM_ITEM_TYPES_MAP map. FORM_ITEM_TYPES_MAP stores the mapping between component types and form item types.

[0140] Construct a SchemaObject object: Use the SchemaObject's builder mode to start building the SchemaObject object.

[0141] Extract the value of the "name" field from the formItem and set it as the name attribute of the SchemaObject.

[0142] Call the getBsonType() method of formItemType to obtain the corresponding BSON type and set it as the bsonType attribute of SchemaObject.

[0143] Set the componentType obtained previously as the component property of the SchemaObject.

[0144] Call the build() method to complete the construction of the SchemaObject object.

[0145] Return result: Returns the constructed SchemaObject object.

[0146] Technical features Component mapping: This implements the mapping from front-end form components (represented as JSONObject) to database table structure objects (SchemaObject). Through the FORM_ITEM_TYPES_MAP mapping, the corresponding form item type can be found based on the form component type, and the bsonType property of the SchemaObject can be determined.

[0147] Builder pattern: Using the SchemaObject Builder pattern to create SchemaObject objects makes the object creation process clearer and more flexible. You can easily set various properties of the SchemaObject, and the code is highly readable.

[0148] Data extraction: Extract required field values, such as "name" and "component", from the formItem JSONObject and use them to construct a SchemaObject object, thus achieving effective data utilization and conversion.

[0149] This method receives a JSONObject object representing the form component and returns a SchemaObject object, which implements the mapping of the front-end form component to the database field.

[0150] ③Data operation interface Implementation method: Provide a unified data addition, deletion, modification and query operation interface, and provide corresponding buttons and forms in the visual interface to allow users to easily perform data addition, deletion, modification and query operations.

[0151] ④Statistical analysis function Implementation method: Built-in statistical analysis query builder supports complex data analysis requirements and provides a query builder in the visual interface, allowing users to build statistical analysis queries in a graphical way.

[0152] The logical processing is described as follows: Get component type: Extract the value of the "component" field from the passed formItem (JSONObject representing form component information) and store it in the componentType variable as the type of the form component.

[0153] Find the configuration corresponding to the form component type: Use componentType as the key and find the corresponding FormItemType object from the FORM_ITEM_TYPES_MAP map. This object contains some configuration information corresponding to the component type, such as bsonType.

[0154] Construct a SchemaObject object: Call the SchemaObject.builder() method to create a SchemaObject builder instance.

[0155] Set the properties of SchemaObject in sequence through the builder: Get the value of the "name" field from formItem and set it as the name attribute of SchemaObject.

[0156] Call the getBsonType() method of formItemType to get the corresponding BSON type, and set it as the bsonType property of SchemaObject.

[0157] Set the previously obtained componentType as the component property of SchemaObject.

[0158] Call the build() method of the builder to complete the construction of the SchemaObject object.

[0159] Return the result: return the constructed SchemaObject object as the return value of the method.

[0160] Technical features Component mapping mechanism: Through FORM_ITEM_TYPES_MAP, the mapping from the front-end form component type to the FormItemType configuration is realized, making it possible to obtain the corresponding configuration information according to different form component types, and then provide the necessary data for building SchemaObject.

[0161] Builder pattern: Use the builder pattern to create SchemaObject objects, making the setting process of object properties more clear and flexible. It is convenient to set each property of SchemaObject individually, and more properties can be easily extended when needed.

[0162] Data conversion: Convert the information of the front-end form component (stored in formItem) into a database table structure object (SchemaObject), realizing the data mapping from the front-end to the database level, and facilitating the subsequent database operation.

[0163] Further, the system also supports the following functions: Dynamic data source routing, dynamically switch to the corresponding data source according to the data source information in DbCommand; Data operation execution, parse and execute JSON format instructions through CommandExecutor, support data add, delete, modify and query operations; Data integration capability, support multi-table joint query and complex data integration operation across data sources; Data security, built-in data encryption and permission management functions to ensure data security.

[0164] It should be noted that: Dynamic data source routing Step 1: Obtain data source information Extracting data source-related information from the DbCommand object is the basis for dynamically switching data sources. DbCommand serves as a unified data operation instruction carrier, containing various types of information required for execution, including the identification of the data source.

[0165] Step 2: Dynamically switch data sources Based on the extracted data source information, DynamicRoutingDataSource is used to achieve dynamic switching. DynamicRoutingDataSource can direct system operations to the corresponding data source based on different data source identifiers, thereby enabling flexible access to different data sources.

[0166] Data operation execution Step 1: Parse JSON instructions Use CommandParser to parse JSON format instructions and convert them into DbCommand objects. CommandParser is a tool specifically designed to parse JSON instructions, which can convert complex JSON data into DbCommand objects that the system can recognize and process.

[0167] Step 2: Execute data operations Execute the parsed DbCommand objects through CommandExecutor to complete data insertion, deletion, modification, and query operations. CommandExecutor is responsible for uniformly executing various database operation instructions and, based on the operation type (such as insertion, deletion, modification, or query) in DbCommand, calls the corresponding method to implement specific data operations.

[0168] Step 3: Support composite instructions The system not only supports basic data operations but also supports composite instructions, such as multi-table join and batch operations. For these complex operations, CommandExecutor will execute each sub-operation in the logical order of the instructions to achieve complete business requirements.

[0169] Data integration capabilities Step 1: Multi-table join implementation The system supports multi-table join queries, meaning you can retrieve data from multiple database tables in a single operation. When performing multi-table join queries, CommandExecutor generates corresponding SQL statements based on the conditions and relationships in the command, implementing the join query across multiple tables through JOOQ. JOOQ automatically identifies database dialects and generates SQL statements adapted to different databases, ensuring query accuracy and efficiency.

[0170] Step 2: Data integration across data sources For complex data integration operations across multiple data sources, the system uses dynamic data source routing to switch between them. During operations, data is extracted from various sources and integrated according to the instructions. Furthermore, the system supports distributed transaction processing across data sources to ensure data consistency and integrity.

[0171] Data security Step 1: Data encryption The system features built-in data encryption, encrypting sensitive information during storage and transmission. Key data fields are encrypted during storage to prevent unauthorized access and tampering within the storage medium. Data is encrypted using secure transmission protocols to ensure data security during network transmission.

[0172] Step 2: Permission Management Through the permissions management function, the operational permissions of different users or roles are strictly controlled. The system assigns specific permissions to each user or role, and only users with corresponding permissions can operate on specific data. Before performing data operations, the system verifies the user's permissions. Only after the permissions are verified will the corresponding operation be allowed, effectively preventing unauthorized access and manipulation of data.

[0173] In addition, in order to better illustrate the technical solution of the present invention, a method for implementing multi-source heterogeneous data integration on a low-code platform is provided. The method adopts the multi-source heterogeneous data integration system on the low-code platform described above, and includes the following steps: Data source configuration and management: Initialize the dynamic data source manager through the DataSourceConfig configuration class, supporting dynamic configuration of multiple database types; Unified command parsing and execution: parse JSON format commands through CommandParser and execute data operations through CommandExecutor; Dynamic data source routing: Dynamically switch to the corresponding data source based on the data source information in DbCommand; JOOQ unified data access: unified access to multiple databases is implemented through JOOQ, which automatically identifies database dialects and generates corresponding SQL statements; Data operation execution: according to the operation type, the corresponding database operation is executed, including data insertion, deletion, modification and query operations.

[0174] Further, the data source configuration and management step includes: Initialize the dynamic data source manager; Select the appropriate data source creator according to the database driver type; Support the configuration of default data sources and extended data sources.

[0175] Further, the unified instruction parsing and execution step includes: Parse the JSON format instruction into a DbCommand object; Execute the data operation instruction, support data insertion, deletion, modification and query operations; Support composite instructions, including multi-table query and batch operations.

[0176] Further, the JOOQ unified data access step includes: Obtain database metadata, identify database type and version; Select the corresponding SQL dialect according to the database type; Generate the corresponding SQL statement through JOOQ and execute it.

[0177] The following best embodiments are described in detail in combination with the drawings: Example 1: Multi-data source unified access implementation Step 1: Data source configuration and management When the system starts, initialize the dynamic data source manager through the DataSourceConfig configuration class: The logical processing is described as follows: Dynamic routing data source configuration: The dynamicRoutingDataSource method is responsible for creating and configuring the dynamic routing data source.

[0178] First, create a DynamicRoutingDataSource instance, pass in the providers object, which may contain information related to the data source.

[0179] Call the addDataSource method to add the default data source to the dynamic routing data source, and the default data source is created by the primaryDataSource method.

[0180] Finally, the configured dynamic routing data source instance is returned.

[0181] Primary data source configuration: The primaryDataSource method is responsible for creating the primary data source.

[0182] Create a DataSourceProperty instance and set various properties of the data source, such as the connection pool name, driver class name, URL, user name, and password.

[0183] Use the EnhancedDataBaseDialect.isRelationalByDriver method to determine whether the database driver type is a relational database.

[0184] If it is a relational database, use hikariDataSourceCreator to create a data source and return it.

[0185] If the database type is not supported, a RuntimeException is thrown.

[0186] Technical features Spring configuration class: Use the @Configuration annotation to mark the DataSourceConfig class as a Spring configuration class, which is used to configure the data source in the Spring application.

[0187] Dynamic routing data source: Dynamic switching of data sources is achieved through DynamicRoutingDataSource, which can dynamically select the appropriate data source according to different requests, improving the flexibility and scalability of the system.

[0188] Data source creation: Select the appropriate data source creator according to the database driver type, and use hikariDataSourceCreator to create a data source for a relational database, which reflects the support and adaptation for different database types.

[0189] Exception handling: When encountering an unsupported database type, a RuntimeException is thrown to ensure that the system can promptly feedback error information when encountering an unsupported situation.

[0190] Step 2: Unify instruction parsing and execution When a user initiates a data operation request through the web interface, the system parses the JSON format command through CommandParser: The logical processing is described as follows: executeRequest method Get JSON command: Get the command in JSON format from the incoming CommandRequest object.

[0191] Parse the command: Call the parseCommand method to parse the obtained JSON command into a DbCommand object.

[0192] Execute the command: Call the executeCommandRequest method to execute the parsed DbCommand object and return the execution result.

[0193] parseCommand method Parse JSON instructions: Create a CommandParser instance and call its parseCommand method to parse the incoming JSONObject type instruction to obtain a DbOperation object.

[0194] Create DbCommand Object: Create a new DbCommand object.

[0195] Execute the operation: Call the execute method of the DbOperation object, pass in the DbCommand object, and perform the operation on the DbCommand object.

[0196] Return result: Returns the processed DbCommand object.

[0197] Technical features Separation of instruction parsing and execution: Separate the parsing and execution processes of JSON instructions. The parseCommand method is responsible for parsing JSON instructions into DbCommand objects, and the executeRequest method is responsible for executing the parsed DbCommand objects, which improves the maintainability and scalability of the code.

[0198] Encapsulation: The specific parsing and operation logic is encapsulated through the CommandParser class and the DbOperation class, making the code of the CommandExecutor class more concise and facilitating the modification and extension of the parsing and operation logic.

[0199] Unified instruction carrier: Use the DbCommand object as a unified data operation instruction carrier to facilitate the transmission and processing of instructions in the system, so that different operations can be performed based on the same object.

[0200] Step 3: Dynamic Data Source Routing The system dynamically switches to the corresponding data source based on the data source information in DbCommand: The logical processing is described as follows: Get data source key: Extract the data source key from the dbCommand object. This key is used to identify a specific data source.

[0201] Switching the data source context: Call the push method of DynamicDataSourceContextHolderProxy and pass in the acquired data source key to dynamically switch the current data source context. Subsequent database operations will then be performed based on this new data source context.

[0202] Update the data source information of DbCommand: Set the obtained data source key back to the dbCommand object to ensure that the data source key recorded in the dbCommand object is the latest.

[0203] Through dataSourceManager, obtain the corresponding data source type according to the data source key, and then set this data source type to the dbCommand object so that dbCommand contains complete data source related information.

[0204] Technical features Dynamic data source switching: DynamicDataSourceContextHolderProxy enables dynamic switching of data source contexts. At runtime, based on the data source key in the dbCommand, you can flexibly switch to different data sources, meeting the needs of multiple data source environments.

[0205] Data consistency: When switching the data source context, the data source key and data source type information in the dbCommand object are updated, ensuring the consistency and accuracy of the data within the dbCommand object, so that subsequent database operations can be performed based on the correct data source information.

[0206] Data source type management: Manage and obtain data source types through the dataSourceManager, encapsulate the data source type acquisition logic in the dataSourceManager, improve the maintainability and scalability of the code. If you need to modify the data source type acquisition logic, you only need to adjust it in the dataSourceManager.

[0207] Step 4: JOOQ unified data access Use JOOQ as a unified data access abstraction layer to automatically identify database dialects: Logical processing description: Try to get database metadata: Get the metadata of the database from the incoming database connection object through the connection.getMetaData() method, which contains information such as database product name and version.

[0208] Determine the database type and version: Extract the database product name and version number from the metadata and store them in the databaseType and databaseVersion variables, respectively.

[0209] Get SQL dialect: Call the getSQLDialect method to determine the SQL dialect used by the database based on the database type and version number.

[0210] Create a configuration object: Create a DefaultConfiguration object to configure the relevant parameters for JOOQ to interact with the database.

[0211] Configure connection, dialect, and log listener: Set the database connection, SQL dialect, and log listener provider to the configuration object in turn. The log listener can record the SQL statements executed by JOOQ, facilitating debugging and monitoring.

[0212] Create and return DSLContext object: Use the configured DefaultConfiguration object to create a DSLContext object through the DSL.using(configuration) method and return it.

[0213] Exception handling: If an SQLException exception occurs during the acquisition of database metadata, catch the exception and throw a RuntimeException exception containing error information, indicating that the database dialect acquisition failed.

[0214] Technical feature points Automatic database dialect identification: By obtaining database metadata, product name, and version number, it automatically identifies the database type and version, and then determines the corresponding SQL dialect, achieving unified access to multiple databases.

[0215] Use of configuration objects: Use the DefaultConfiguration object to centrally manage the configuration of JOOQ's interaction with the database, including connections, dialects, and log listeners, which improves the maintainability and flexibility of the code.

[0216] Logging function: By setting DefaultExecuteListenerProvider and JooqSqlLogger, you can log the SQL statements executed by JOOQ, making it easier for developers to debug and monitor database operations.

[0217] Exception handling: Exception handling is performed when obtaining database metadata. When a SQLException occurs, a RuntimeException containing error information is thrown, which enhances the robustness of the code.

[0218] Unified data access interface: The returned DSLContext object provides developers with a unified interface, shielding the differences between different databases, allowing developers to perform SQL operations on different databases in the same way.

[0219] Step 5: Data Operation Execution Execute the corresponding database operation according to the operation type: The logical processing is described as follows: Get operation type: Get the operation type from the passed-in DbCommand object. The operation type is used to determine which database operation needs to be performed.

[0220] Operation type judgment: Use the switch statement to judge the operation type and execute the corresponding processing logic according to different operation types.

[0221] Perform the corresponding operations: If the operation type is METHOD_GET, the executeGetCommand method is called to execute the query operation and return the operation result.

[0222] If the operation type is METHOD_ADD, the executeAddCommand method is called to perform the add operation and return the operation result.

[0223] If the operation type is METHOD_UPDATE, the executeUpdateCommand method is called to perform the update operation and return the operation result.

[0224] If the operation type is METHOD_REMOVE, the executeRemoveCommand method is called to perform the delete operation and return the operation result.

[0225] Exception handling: If the operation type does not match any defined type (that is, it is not within the range of METHOD_GET, METHOD_ADD, METHOD_UPDATE, METHOD_REMOVE), an UnsupportedOperationException is thrown, indicating that the operation type is not supported.

[0226] Technical features Unified command execution: The executeDbCommand method provides a unified entry point for processing different types of database operation commands, centrally managing command parsing and execution logic, and improving code maintainability and scalability.

[0227] Operation type separation: Use a switch statement to call different processing methods based on different operation types, making the logic of different operations independent of each other and facilitating code modification and expansion. If you need to add a new operation type, simply add a new case branch to the switch statement.

[0228] Encapsulation: Encapsulating the specific operation logic in methods such as executeGetCommand, executeAddCommand, executeUpdateCommand, and executeRemoveCommand makes the code of the executeDbCommand method concise and clear, and also improves the reusability of the code.

[0229] Exception handling: Exception handling for unsupported operation types is performed through the default branch, which enhances the robustness of the code and avoids program crashes caused by the introduction of unknown operation types.

[0230] Example 2: Visual table structure design Step 1: Form component mapping The system provides a visual form design interface that supports mapping of multiple component types to database fields: The logical processing is described as follows: Get component type: Extract the value of the "component" field from the passed formItem (JSONObject representing form component information) and store it in the componentType variable as the type of the form component.

[0231] Find the configuration corresponding to the form component type: Use componentType as the key and find the corresponding FormItemType object from the FORM_ITEM_TYPES_MAP map. This object contains some configuration information corresponding to the component type, such as bsonType.

[0232] Construct a SchemaObject object: Call the SchemaObject.builder() method to create a SchemaObject builder instance.

[0233] Set the properties of SchemaObject in sequence through the builder: Get the value of the "name" field from formItem and set it as the name attribute of SchemaObject.

[0234] Call the getBsonType() method of formItemType to obtain the corresponding BSON type and set it as the bsonType attribute of SchemaObject.

[0235] Set the componentType obtained previously as the component property of the SchemaObject.

[0236] Get the value of the "title" field from formItem and set it as the title attribute of SchemaObject.

[0237] Get the Boolean value of the "required" field from the formItem and set it as the required attribute of the SchemaObject.

[0238] Call the builder's build() method to complete the construction of the SchemaObject object.

[0239] Return result: The constructed SchemaObject object is used as the return value of the method.

[0240] Technical features Component mapping mechanism: The mapping from the front-end form component type to the FormItemType configuration is implemented through FORM_ITEM_TYPES_MAP, so that the corresponding configuration information can be obtained according to different form component types, thereby providing the necessary data for building SchemaObject.

[0241] Builder pattern: Using the Builder pattern to create a SchemaObject object makes the process of setting object properties clearer and more flexible. You can easily set each property of the SchemaObject individually, and you can easily expand the setting of more properties when needed.

[0242] Data conversion: Convert the information of the front-end form component (stored in formItem) into a database table structure object (SchemaObject), realizing data mapping from the front-end to the database level, facilitating subsequent database operations.

[0243] Attribute integrity: When constructing a SchemaObject, not only the basic name, bsonType, and component attributes are set, but also the title and required attributes are set, ensuring that the generated SchemaObject contains more complete form component information, making the database table structure object more in line with business needs.

[0244] Step 2: Dynamic table structure management The system supports dynamic creation and modification of table structures: The logical processing is described as follows: Get the data source: According to the data source name in the TableDO object, call the JooqBaseUtil.getDataSourceByName method to get the corresponding data source.

[0245] Get the database connection: Use the DataSourceUtils.getConnection method to get the database connection from the data source.

[0246] Try to create the table: Through the JooqBaseUtil.getDslContext method, a DSLContext object is created based on the database connection. This object is used to perform SQL operations.

[0247] Call the JooqDDLUtil.generateCreateTableDsl method and combine the DSLContext and TableDO objects to generate the DSL steps for creating a table.

[0248] Use the execute method of DSLContext to execute the DSL step of creating the table.

[0249] If the execution is successful, return true.

[0250] Release the connection: Regardless of whether the table creation operation is successful, call the DataSourceUtils.releaseConnection method in the finally block to release the database connection to ensure that resources are released correctly.

[0251] Technical features Data source management: Get the data source according to the data source name through JooqBaseUtil, realize the dynamic acquisition and management of data source, and facilitate switching between different data sources.

[0252] Database connection management: Use DataSourceUtils to obtain and release database connections, ensuring the correct acquisition and release of database connections and avoiding resource leaks.

[0253] JOOQ Integration: Leverage JOOQ's DSLContext object to execute SQL operations and generate table creation DSL steps using JooqDDLUtil, enabling dynamic database table creation using JOOQ. JOOQ can automatically identify database dialects and generate corresponding SQL statements, improving code portability.

[0254] Exception handling and resource release: Use the try-finally structure to ensure that the database connection is released regardless of whether the table creation operation is successful, thereby enhancing the robustness of the code.

[0255] Example 3: Unified API interface Step 1: Unified Web Interface The system provides a unified Web interface that supports various data operations: The logical processing is described as follows: Record the start time: At the beginning of the method, call System.currentTimeMillis() to record the current time as the start time of the operation.

[0256] Execute request processing: pass the received CommandRequest object to the executeRequest method of commandExecutor for processing and obtain the processing result result.

[0257] Calculate execution time: Call System.currentTimeMillis() again to get the current time, subtract it from the start time, and get the execution time of the operation.

[0258] Record execution time logs: Use logging tools (such as log.info) to record the execution time of operations to facilitate subsequent performance monitoring and analysis.

[0259] Return result: Return the processing result to the caller.

[0260] Technical features Spring MVC Annotation: Use the @PostMapping("web") annotation to map this method to a POST request processing method with the path being web, which complies with the Spring MVC development model and facilitates the construction of RESTful APIs.

[0261] Request processing encapsulation: The specific request processing logic is encapsulated through the commandExecutor.executeRequest method, making the web method code more concise and improving the maintainability and scalability of the code.

[0262] Performance Monitoring: By recording the start and end times of operations and calculating and recording execution time, we can help monitor and optimize system performance. We can use execution time to determine request processing efficiency and identify performance bottlenecks.

[0263] Logging: Use logging tools to record execution time, making it easier for developers and operations personnel to view and analyze system performance. Logs can help locate problems, optimize performance, and conduct audits.

[0264] Step 2: Dynamic API call The system supports dynamic API call function: The logical processing is described as follows: Find API information: Use the findById method to find the corresponding ApiDO object based on the passed id. The object contains the relevant information of the API.

[0265] Check the existence of the API: Use Assert.notNull to assert and ensure that the found ApiDO object is not null. If it is null, an exception is thrown, indicating that the API interface does not exist.

[0266] Set the interface package configuration: call the setApiPackage method to set the interface package configuration according to the ApiDO object.

[0267] Build the request URL: Call the ApiCommonUtil.buildApiUrl method to build the request URL based on the ApiDO object, request parameters params, and environment information env.

[0268] Build request parameters: Call the ApiRequestParamUtil.buildUrl method to build request parameters based on the ApiDO object, request parameters params, request URL apiUrl, and system context systemContext.

[0269] Build the request body: Call the ApiRequestBodyUtil.buildRequestBody method to build the request body based on the ApiDO object, request body, request parameter builder urlBuilder, and system context systemContext.

[0270] Execute HTTP request: call ApiRequestBuilderUtil.sendHttp method, pass in ApiDO object, request parameter builder urlBuilder and request body requestBody to execute HTTP request and return the request result.

[0271] Technical features API management: By searching for API information by ID, unified management and calling of APIs are achieved, making it easier to configure and use different APIs.

[0272] Parameter validation: Use Assert.notNull to perform parameter validation to ensure the existence of API information, avoid subsequent errors caused by the non-existence of the API, and enhance the robustness of the code.

[0273] Modular design: Encapsulating URL construction, request parameter construction, request body construction, and HTTP request execution into separate tool classes improves code maintainability and scalability. Different tool classes are responsible for different functions, making them easy to modify and extend.

[0274] Unified calling interface: The callApi method provides a unified interface to call the API, shielding the underlying implementation details and making the API calling code more concise and clear.

[0275] Example 4: Multi-table join query and data integration Step 1: Implement cross-data source join query The system supports join table queries across different data sources under the same database server: The logical processing is described as follows: Obtain data source and connection: According to the data source key in the DbCommand object, obtain the corresponding data source through the JooqBaseUtil.getDataSourceByKey method, and then use the DataSourceUtils.getConnection method to obtain a database connection from the data source.

[0276] Obtain DSLContext object: Use the JooqBaseUtil.getDslContext method to create a DSLContext object based on the database connection, which is used to execute SQL operations.

[0277] Obtain table set and association: Call the JoinTable.getTableJoin method to obtain the table set participating in the multi-table join and their association based on the DbCommand object.

[0278] Build query object: Use dslContext.selectQuery() to create a SelectQuery object to build the SQL statement for multi-table join.

[0279] Add main table: Get the name of the first table from the DbCommand object, use DSL.table method to build the main table object, and add it to SelectQuery.

[0280] Add associated tables and JOIN conditions: Traverse the associated table list, for each associated table, use DSL.table method to build the associated table object, and add the associated table and corresponding JOIN conditions to SelectQuery.

[0281] Execute query and return result: Call selectQuery.fetch().intoMaps() method to execute the query, and convert the query result to List<Map<String, Object>> type and return.

[0282] Release connection: Regardless of whether the query operation is successful, call DataSourceUtils.releaseConnection method in the finally block to release the database connection, ensuring that resources are properly released.

[0283] Technical feature points Data source management: Dynamically obtain data sources through data source keys, support for different data sources, and facilitate operations in a multi-data source environment.

[0284] JOOQ integration: Use JOOQ's DSLContext and SelectQuery to build and execute SQL queries. JOOQ can automatically identify database dialects and generate SQL statements adapted to different databases, improving code portability.

[0285] Multi-table join query: By obtaining table sets and association relationships, dynamically construct multi-table join query SQL statements, and support obtaining data from multiple database tables in one operation.

[0286] Resource management: Use the try-finally structure to ensure that the database connection is released after use, avoiding resource leaks and enhancing the robustness of the code.

[0287] Result conversion: convert query results into List <Map<String, Object> > type, which facilitates the processing and use of query results.

[0288] Step 2: Statistical analysis of data The system has built-in statistical analysis query builder: The logical processing is described as follows: Get the data source table name: Get the actual table name from the TableDO object, and then build the corresponding Table object through the JooqBaseUtil.buildDslTable method for subsequent queries.

[0289] Build query conditions: Get the data source rule list from the StatisticsDTO object, combine it with a new JSONObject, and use the JooqConditionUtil.conditionsToJooqCondition method to convert the rules into a JOOQ Condition object as the query filter condition.

[0290] Build group fields: Call the fieldProcessor.buildGroupFields method to build a group field list based on the StatisticsDTO and TableDO objects.

[0291] Build summary fields: Call the fieldProcessor.buildSummaryFields method to build a summary field list based on the StatisticsDTO and TableDO objects.

[0292] Build a statistical query: Use the DSLContext object to build a query, select grouping fields and summary fields, specify the query table, set query conditions, and group by grouping fields.

[0293] Execute the query and return the result: Execute the constructed query and convert the query result into a List <Map<String,Object> > type and return.

[0294] Technical features JOOQ integration: Use JOOQ's DSLContext to build and execute SQL queries, which can automatically identify database dialects and generate SQL statements adapted to different databases, improving code portability.

[0295] Dynamic query construction: Dynamically build query conditions, grouping fields, and summary fields based on StatisticsDTO and TableDO objects, so that the query can be flexibly changed according to different configurations.

[0296] Modular design: Encapsulates functions such as table name construction, query condition construction, group field construction, and summary field construction in different tool classes and methods, improving the maintainability and scalability of the code.

[0297] Result conversion: convert query results into List <Map<String, Object> > type, which facilitates the processing and use of query results.

[0298] Through the above embodiments, the present invention can effectively achieve unified integration and management of multi-source heterogeneous data, providing an efficient and flexible solution for enterprise-level application development. The system has good scalability and maintainability, and can adapt to the data integration needs of different enterprises.

[0299] The multi-source heterogeneous data integration system of the present invention has the following technical features: Advanced architecture: adopts layered architecture design, with clear responsibilities of each module, easy to maintain and expand Technology maturity: Based on mature open source technology stack, stable and reliable Operational uniformity: Provides a unified data operation interface to shield the differences in the underlying database Development efficiency: Visual development interface greatly improves development efficiency Expansion flexibility: supports rapid integration and expansion of new database types The system has been verified in actual projects, proving the feasibility and practicality of its technical solution.

[0300] Additionally, to verify the beneficial effects of the present application, the following technical application examples are provided: Technical Application Example 1: Cross-warehouse order-inventory join and real-time alert Scenario An operator wants to view order data from Oracle and inventory data from MySQL in a single visual interface, and receive real-time alerts when inventory is below a safety threshold.

[0301] Input Collection • Select in the low-code platform drag-and-drop configurator: • Datasource A: oracle_sales (order table T_ORDER) • Datasource B: mysql_inventory (inventory table T_STOCK) Configure JSON instructions: { "op": "GET", "tables": [ {"name":"oracle_sales.T_ORDER","alias":"o"}, {"name":"mysql_inventory.T_STOCK","alias":"s","join":"o.sku_id =s.sku_id"} ], "fields":["o.order_id","o.sku_id","o.qty","s.stock_qty"], "where":[{"field":"s.stock_qty","op":"<=","value":50}] } Preprocessing 1. Dynamic data source detection: The dynamic data source management module detects the connection status of Oracle and MySQL and puts them into the connection pool.

[0302] 2. Schema pulling: The unified data access abstraction layer pulls the table structures of the two databases for field type comparison.

[0303] Parsing and execution 1. CommandParser converts JSON instructions into DbCommand objects.

[0304] 2. DynamicRoutingDataSource splits the query based on the tables element and opens two connections within the same JVM; TransactionManager creates a cross-database transaction.

[0305] 3. JOOQ automatically identifies Oracle and MySQL dialects and generates database JOIN SQL; CommandExecutor executes and merges the results.

[0306] 4. The results are rendered in real time in the visual dashboard component; when the stock_qty of any row is ≤ 50, the alert rule engine triggers a WebSocket notification.

[0307] Output • Front-end table and inventory balance heat map.

[0308] • Instant pop-up + email reminder “SKU {X} is out of stock”.

[0309] Technology Application Example 2: Drag-and-drop table structure expansion and automatic code synchronization Scenario The business analyst adds a new "loyalty_level" field to the "customer information" data model and wants to make it immediately available in the CRUD pages and REST API.

[0310] Input Collection • Drag the “Drop-Down Box” component to the form in the visual model designer; set: • Field name: loyalty_level • Component type: select • Options: Bronze / Silver / Gold / Platinum Preprocessing 1. convertFormItemToSchemaItem maps the form component to a SchemaObject (bsonType=string).

[0311] 2. ModelDiffService compares the original table schema and generates incremental DDL.

[0312] Parsing and execution 1. TableService.createOrAlterTable constructs ALTER TABLE ADD COLUMN loyalty_level VARCHAR(32) by JooqDDLUtil and hot updates in the PostgreSQL database.

[0313] 2. MetaCodeGenerator listens to schema update events and automatically regenerates: • MyBatis-plus Mapper + Entity • Spring Controller's CRUD interfaces • Form, table columns, and validation rules in the frontend Vue3 components 3. HotReloadProxy injects new code packages into running microservices without restarting.

[0314] Output • The new field is immediately displayed on the business page; users can immediately enter loyalty information.

[0315] • The REST / api / customer adds the loyaltyLevel field, and the OpenAPI document is updated in real time.

[0316] Technical Application Example 3: One-Click Multi-Dimensional Statistical Report Generation Scenario The finance department needs to generate an interactive report on "Sales in Each Region vs. Budget Completion Rate" for the current quarter and allow drilling by month and region.

[0317] Input Collection • Select the data source mysql_finance, table T_SALES, and table T_BUDGET in the report wizard.

[0318] • Select dimensions region and month and measures actual_amount and budget_amount by dragging and dropping, and check "Automatically calculate completion rate".

[0319] Preprocessing 1. The low-code report builder converts the drag-and-drop operation into StatisticsDTO configuration: { "group":["region","month"], "summary":[ {"field":"actual_amount","agg":"SUM"}, {"field":"budget_amount","agg":"SUM"}, {"expr":"SUM(actual_amount) / NULLIF(SUM(budget_amount),0)","alias":"rate"} ] } 2. StatisticMetaValidator checks the validity of fields and aggregation functions.

[0320] Parsing and execution 1. StatisticsQueryBuilder.buildAndExecuteQuery generates group statistics SQL, and JOOQ automatically infers the dialect.

[0321] 2. ResultSetTransformer converts the query results into a multidimensional cube (OLAP cubeformat).

[0322] 3. ChartService calls the built-in ECharts configuration template to generate an interactive line-bar hybrid chart; at the same time, it generates the REST / report / {id} / data interface.

[0323] Output • Browser-side rendering: • Bars represent actual sales • The line represents the budget completion rate • The user can drill down to the detailed order list by clicking on any column, and the system returns the data through the same DbCommand deep query.

[0324] The present invention provides a low-code platform multi-source heterogeneous data integration system, which has the following beneficial effects: 1. Unity Through a unified data access abstraction layer, unified operations on multiple heterogeneous databases are achieved, and developers do not need to worry about the specific types and differences of the underlying databases.

[0325] 2. Efficiency The use of dynamic data source routing and connection pool management technology improves database access efficiency and reduces connection overhead.

[0326] 3. Flexibility Unified instruction system supports complex data operation requirements, and various database operations can be achieved through JSON configuration without writing code.

[0327] 4. Scalability Modular design makes the system easy to expand and supports the rapid integration of new database types.

[0328] 5. Development efficiency Visual development interface greatly reduces the technical threshold of application development and improves development efficiency.

[0329] 6. Cost savings Reduces repetitive development work and reduces system maintenance costs.

[0330] 7. Data security Built-in data encryption and permission management functions ensure data security.

[0331] Solves the problem of how to build a unified multi-source heterogeneous data integration system and method, realizes unified access and operation of different types of databases, and provides visual low-code development capability, reduces the complexity and cost of enterprise-level application development.

[0332] It should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A low-code platform multi-source heterogeneous data integration system, characterized by: The system includes the following modules: Dynamic data source management module, used to support dynamic configuration and management of multiple database types, including data source configuration management, dynamic data source routing, connection pool management, and data source health check; Unified data access abstraction layer, based on JOOQ, enables unified access to multiple databases, including automatic SQL dialect recognition, data type mapping, and cross-data source transaction management; A unified instruction system for parsing and executing data operation instructions in JSON format, including instruction parser, instruction executor, instruction encapsulation, and composite instruction support for complex data operations; The visual development module provides a visual table structure design interface, form component mapping, data operation interface, and statistical analysis functions.

2. The low-code platform multi-source heterogeneous data integration system according to claim 1 is characterized in that: The dynamic data source management module includes: Dynamic data source routing, based on DynamicRoutingDataSource to achieve dynamic switching of data sources; Connection pool management, using Hikari connection pool technology to implement database connection management; Data source health check, providing data source connection testing and status monitoring functions.

3. The low-code platform multi-source heterogeneous data integration system according to claim 2 is characterized in that: The unified data access abstraction layer includes: JOOQ integration: unified access to multiple databases through JOOQ, automatic identification of database dialects and generation of corresponding SQL statements; Data type mapping: provides a unified data type mapping mechanism to shield data type differences between different databases; Transaction management, supporting distributed transaction processing across data sources.

4. The low-code platform multi-source heterogeneous data integration system according to claim 3 is characterized in that: The unified instruction system includes: Instruction parser, used to parse data operation instructions in JSON format; The instruction executor uniformly executes various database operation instructions, including data addition, deletion, modification and query operations; Instruction encapsulation, using the DbCommand object as a unified data operation instruction carrier; Compound operations support, including multi-table joint query and batch operation compound instructions.

5. The low-code platform multi-source heterogeneous data integration system according to claim 4 is characterized in that: The visual development module includes: Table structure design interface, supporting configuration of field types and constraints; Form component mapping, mapping front-end form components to database fields, supporting multiple component types; Data operation interface, providing a unified data addition, deletion, modification and query operation interface; Statistical analysis function, built-in statistical analysis query builder, supports data analysis needs.

6. The low-code platform multi-source heterogeneous data integration system according to claim 5 is characterized in that: The system also supports the following features: Dynamic data source routing, dynamically switching to the corresponding data source according to the data source information in DbCommand; Data operation execution: CommandExecutor parses and executes JSON format instructions, supporting data addition, deletion, modification and query operations; Data integration capabilities, supporting multi-table join queries and cross-data source data integration operations; Data security, built-in data encryption and permission management functions to ensure data security.

7. A method for implementing multi-source heterogeneous data integration on a low-code platform, using the multi-source heterogeneous data integration system on a low-code platform as described in any one of claims 1 to 6, characterized in that: The method comprises the following steps: Data source configuration and management: Initialize the dynamic data source manager through the DataSourceConfig configuration class, supporting dynamic configuration of multiple database types; Unified command parsing and execution: parse JSON format commands through CommandParser and execute data operations through CommandExecutor; Dynamic data source routing: Dynamically switch to the corresponding data source based on the data source information in DbCommand; JOOQ unified data access: JOOQ enables unified access to multiple databases, automatically identifying database dialects and generating corresponding SQL statements; Data operation execution: Execute corresponding database operations according to the operation type, including data addition, deletion, modification and query operations.

8. The method for implementing multi-source heterogeneous data integration on a low-code platform according to claim 7 is characterized in that: The data source configuration and management steps include: Initialize the dynamic data source manager; Select the data source creator based on the database driver type; Supports configuration of default data sources and extended data sources.

9. The method for realizing multi-source heterogeneous data integration on a low-code platform according to claim 8 is characterized in that: The unified instruction parsing and execution steps include: Parse the JSON format command into a DbCommand object; Execute data operation instructions and support data addition, deletion, modification and query operations; Supports compound instructions, including multi-table join queries and batch operations.

10. The method for realizing multi-source heterogeneous data integration on a low-code platform according to claim 9 is characterized in that: The JOOQ unified data access steps include: Obtain database metadata and identify database type and version; Select the corresponding SQL dialect according to the database type; Generate the corresponding SQL statement through JOOQ and execute it.

Images