A multi-source data fusion sharing platform based on large model and its sharing method

Abstract

The present application discloses a multi-source data fusion sharing platform based on a large model and a sharing method thereof. The data source configuration module is responsible for adding and configuring multiple data sources and binding database drivers. The data directory management module supports the creation, modification and search of data directories. The large model processing module uses pre-trained models and vector libraries to generate SQL statements suitable for the target database according to user needs. The API interface management module provides a rich API interface to facilitate user interaction with the platform. The network configuration module ensures communication between the platform and external systems. The service migration module supports the migration and expansion of platform services. The database module is responsible for storing and managing the platform's data, including data source information, data directory information and user information. It solves the problems of the traditional MVC development model in data access being cumbersome, inefficient and difficult to cope with the big data environment.

Description

Technical Field

[0001] The present application relates to the technical field of multi-source data fusion, and in particular to a multi-source data fusion sharing platform based on a large model and a sharing method thereof. Background Art

[0002] In the traditional MVC (Model-View-Controller) development model, the data access process strictly follows the hierarchical structure of the control layer, service layer, and data access layer. Although this structure ensures the modularity and maintainability of the application, it exposes some significant defects in data access.

[0003] In the traditional MVC architecture, each data access request needs to pass through the control layer, service layer, and data access layer in sequence. The control layer is responsible for receiving user requests and forwarding them to the corresponding service layer; the service layer is responsible for processing business logic and calling the data access layer to interact with the database as needed; the data access layer is responsible for performing specific database operations and returning the results to the service layer. Although this layer-by-layer transmission method has a clear structure, it is cumbersome and inefficient in actual operation. Whenever a new data access function needs to be added, developers not only need to modify the code of the data access layer to adapt to the new database operation, but also need to make corresponding adjustments in the service layer and control layer, and may even involve modifications to the front-end page. This method of full code modification is not only time-consuming and labor-intensive, but also increases the risk of software errors, which is not conducive to later maintenance and function upgrades.

[0004] Therefore, how to solve the problems of the traditional MVC development model in data access, which is cumbersome, inefficient and difficult to cope with the big data environment, has become an important issue that needs to be solved urgently. Summary of the invention

[0005] In the embodiments of the present application, by providing a multi-source data fusion and sharing platform based on a large model, the problems of the traditional MVC development model in data access being cumbersome, inefficient and difficult to cope with a big data environment are solved.

[0006] In the first aspect, the embodiment of the present application provides a multi-source data fusion sharing platform based on a big model, including a data source configuration module, a data directory management module, a big model processing module, an API interface management module, a network configuration module, a service migration module and a database module; the data source configuration module is configured to add and configure multiple types of data sources, enter the name, type, URL, user name, password and status of the data source, and add the data source to the data source list, and bind the corresponding database driver; the data directory management module is configured to create, modify and search the data directory, the data directory includes a main directory and sub-directories, supports searching the main directory or sub-directory by keyword, and when a main directory is selected, all sub-directories under the main directory are displayed. table; the large model processing module is configured to receive database operation requirements input by the user, generate SQL statements suitable for the target database by calling the pre-trained large model interface and according to the vector representations of the queries and operation statements in different databases stored in the vector library; the API interface management module is configured to provide API interfaces and manage data items so that users can interact with the platform by calling API interfaces; the network configuration module is configured to configure the network connection parameters of the platform to ensure the normal communication between the platform and external systems; the service migration module is configured to support the migration and expansion of platform services; the database module is configured to store and manage the platform data, including data source information, data directory information and user information.

[0007] In a possible implementation, the large model processing module includes a vector library management unit; the vector library management unit includes a vector generation subunit and a vector storage subunit.

[0008] In the second aspect, the embodiment of the present application provides a sharing method of a multi-source data fusion sharing platform based on a big model. Through the data source configuration module, various types of data sources are added and configured, and the name, type, URL, user name, password and status of the data source are entered so that the platform can connect to the corresponding database system; the configured data source is added to the data source list and bound to the corresponding database driver; the data directory management module is used to collect metadata from the data source and divide the collected metadata into data directories; the creation, modification and deletion of data directories, as well as the binding and unbinding operations of data directories and API interfaces are supported; when the user needs to operate a certain data source, the target data source and the data operation to be performed are selected; wherein the data operation includes data query, addition, modification and deletion operations; the big model processing module is called to perform operations according to the selected target data source. The target data source and the data operations to be performed, as well as the vector representations of the queries and operation statements in different databases stored in the vector library, generate SQL statements suitable for the target database; through the API interface management module, view and manage the API interface list and manage data items, and implement data operations through the API interface; according to user requests, call the database module to execute SQL statements, complete data operations, and return operation results; the network configuration module provides the configuration function of the server's domain name or IP address, supports the front-end page to access through the specified domain name or IP address, and supports the security code verification mechanism to ensure that the front-end page can be authenticated when accessing the database; the service migration module supports migration and expansion between different servers or database systems; the database module stores and manages the data of the platform, including data source information, data directory information and user information.

[0009] In a possible implementation, the calling of the large model processing module generates an SQL statement applicable to the target database according to the selected target data source and the data operation to be performed, as well as the vector representations of the queries and operation statements in different databases stored in the vector library, including: when a user needs to operate on a certain data source, selecting the target data source and the data operation to be performed; calling the large model processing module, selecting the target data source and the data operation to be performed, and searching the vector representations of the queries and operation statements related to the target database operation in the vector library; searching the vector representations of the queries and operation statements related to the target database operation in the vector library, and selecting the target data source and the data operation to be performed are input into the large model to generate an SQL statement applicable to the target database; and the large model performs standardized answer processing on the generated SQL statement so that the returned data format meets the user's requirements.

[0010] In one possible implementation, queries and operation statements in different databases are stored in a vector library in a vector library management unit to serve as a reference resource for the large model when processing database operations; queries and operation statements in different databases are vectorized in a vector generation subunit to generate vector representations; and the generated vector representations are stored in a vector storage subunit to serve as a reference resource for the large model when processing database operations.

[0011] In a possible implementation, the method further includes: optimizing the SQL statements generated by the large model in the large model processing module; the optimization includes index usage optimization, query condition optimization, and connection operation optimization.

[0012] In a possible implementation, it also includes: performing grammatical and semantic analysis on the generated SQL statements in the large model processing module to generate SQL statement explanation information; generating an execution plan for the SQL statements based on the characteristics of the SQL statements, and displaying each step of the execution plan and the estimated execution time to help users understand the execution process of the SQL statements.

[0013] In a possible implementation, it also includes: providing user authority management functions, dividing users into roles and allocating authority; users with different roles have different operation authorities and data access authorities; when a user logs into the platform, menus and buttons of the user interface are dynamically generated according to the user's role and authority information.

[0014] In one possible implementation, when a user submits a data operation request through the API interface management module, the request is received and parsed, and the SQL statement therein is extracted; the legality of the extracted SQL statement is judged according to the preset data operation rules and security policies; if the judgment result is legal, the SQL statement is allowed to continue to execute; if the judgment result is illegal, the execution of the SQL statement is blocked, and an error message is returned to the user, prompting the user to modify or resubmit the data operation request.

[0015] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects:

[0016] The embodiment of the present application provides a multi-source data fusion sharing platform based on a large model. The data source configuration module can add and configure various types of data sources. By entering the basic information of the data source and binding the corresponding database driver, the platform can automatically identify and connect to the data source to achieve rapid data integration. The data directory management module supports the creation, modification and search of data directories, including main directories and subdirectories, which is convenient for users to organize and manage data resources. Through the keyword search function, users can quickly locate the required data directory or table, which improves the efficiency of data search. When the main directory is selected, all tables under the directory are automatically displayed, so that users can intuitively understand the data structure and content. The large model processing module uses the pre-trained large model interface to automatically generate SQL statements suitable for the target database according to the database operation requirements entered by the user. Through the vector representation of different database queries and operation statements stored in the vector library, the large model can accurately understand the user's intentions and generate SQL statements that conform to the syntax of the target database, greatly improving the efficiency and accuracy of database operations. The API interface management module provides a rich API interface, enabling users to interact with the platform by calling these interfaces, such as data query, data modification, data deletion, etc. By managing data items, users can customize the data structure and return format of the API interface to meet different business needs. The network configuration module can flexibly configure the network connection parameters of the platform to ensure normal communication between the platform and external systems. The service migration module supports the migration and expansion of platform services, allowing the platform to be flexibly deployed and expanded according to business needs, reducing operation and maintenance costs. The database module is responsible for storing and managing the platform's data, including data source information, data directory information, and user information, ensuring data integrity and security. It solves the problems of the traditional MVC development model in data access being cumbersome, inefficient, and difficult to cope with the big data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a multi-source data fusion and sharing platform based on a large model provided in an embodiment of the present application;

[0019] Figure 2 A flow chart of a sharing method of a multi-source data fusion sharing platform based on a large model provided in an embodiment of the present application;

[0020] Figure 3A schematic diagram of a large-model-based multi-source data fusion sharing server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0023] At present, different business systems of enterprises are developed at different times, with different business types, and different business systems exist in different database systems. There will be different differences in the syntax of SQL statements in different database systems. The key technology of multi-source data fusion lies in querying and analyzing different database systems. For example, in different database systems such as MySQL, SQL Server, Oracle and ArgoDB, the commands for viewing the database are different. For example, in MySQL, the command to view the database is SHOWDATABASES; in SQL Server, the command to view the database is SELECT name FROMmaster.sys.databases; in Oracle, the command to view the database is SELECT*FROMv$database. Due to the differences in SQL statement syntax between different database systems, traditional data operation methods often require writing specific SQL statements in the code for each database system. This not only increases the development cost, but also reduces the maintainability and flexibility of the system. A multi-source data fusion sharing platform based on a large model and its sharing method proposed in this application generate operation statements for different database systems through a large model, thereby realizing unified data operations on multiple database systems, thereby solving this problem.

[0024] The present application embodiment provides a multi-source data fusion sharing platform based on a large model, such as Figure 1As shown, it includes a data source configuration module, a data directory management module, a large model processing module, an API interface management module, a network configuration module, a service migration module and a database module. The data source configuration module is configured to add and configure various types of data sources, enter the name, type, URL, user name, password and status of the data source, add the data source to the data source list, and bind the corresponding database driver. The data directory management module is configured to create, modify and search the data directory. The data directory includes a main directory and a subdirectory, and supports searching the main directory or subdirectory by keywords. When a main directory is selected, all tables under the main directory are displayed. The large model processing module is configured to receive the database operation requirements input by the user, and generate SQL statements suitable for the target database by calling the pre-trained large model interface and according to the vector representation of the query and operation statements in different databases stored in the vector library. The API interface management module is configured to provide an API interface so that the user can interact with the platform by calling the API interface. The network configuration module is configured to configure the network connection parameters of the platform to ensure the normal communication between the platform and the external system. The service migration module is configured to support the migration and expansion of platform services. The database module is configured to store and manage the platform's data, including data source information, data directory information, and user information.

[0025] The large model processing module includes a vector library management unit, which includes a vector generation subunit and a vector storage subunit.

[0026] The present application embodiment provides a sharing method of a multi-source data fusion sharing platform based on a large model, such as Figure 2 As shown, the method includes steps S101 to S111. Among them, Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a sharing method of a multi-source data fusion sharing platform based on a large model. In the case that the final result can be achieved, Figure 2 The steps shown may be performed in parallel or reversed.

[0027] S101: Through the data source configuration module, add and configure various types of data sources, enter the name, type, URL, user name, password and status of the data source, so that the platform can connect to the corresponding database system.

[0028] Specifically, the data source configuration module supports configuring a variety of common database systems as data sources. These database systems include but are not limited to all versions of MySQL, all versions of SQL Server, all versions of Oracle, all versions of DB2, and all versions of ArgoDB. The data source configuration module is designed to meet the enterprise's integration needs for different database systems and achieve unified management and access to data. When configuring a data source, the user needs to enter the basic information of the data source, including the name and type of the data source and the selected database system type, URL (used to specify the address and port of the database server), user name and password. The user name and password are important credentials for security verification, ensuring that only authorized users can access and operate the database. In addition to entering basic information, the data source configuration function also provides management operations for the data source. Users can enable or disable the configured data source. When a data source is disabled, the platform will not be able to connect to the data source to read or write data. This function provides strong support for flexible management and security control of data sources.

[0029] S102: Add the configured data source to the data source list and bind the corresponding database driver.

[0030] Specifically, in order to establish an actual connection with the database system, the corresponding database driver needs to be bound to the data source. The database driver is a software component that provides the interface and protocol required to communicate with a specific database system. By binding the correct database driver, the platform can establish a stable connection with the database system and execute SQL statements for data operations. In the process of binding the driver, the system will automatically select or prompt the user to select the corresponding database driver according to the type of data source. The user needs to ensure that the selected driver matches the version and type of the database system to ensure the stability and compatibility of the connection.

[0031] S103: Using the data directory management module, metadata is collected from the data source, and the collected metadata is divided into data directories.

[0032] S104: Supports the creation, modification and deletion of data directories, as well as the binding and unbinding of data directories and API interfaces.

[0033] S105: When the user needs to operate a data source, the user selects the target data source and the data operation to be performed, wherein the data operation includes data query, addition, modification and deletion operations.

[0034] Specifically, in the process of data integration and management, the platform first comprehensively collects metadata from various configured data sources through the data catalog management module. These metadata are descriptive information about the data structure in the database system, including but not limited to table names, field names, data types, constraints, etc. In order to ensure the effective use and management of metadata, the platform supports users to divide the collected metadata into data catalogs in a variety of ways. Users can give business-related labels and descriptions to metadata based on the actual business needs and terminology specifications of the enterprise, thereby establishing an association between metadata and business. Users can add custom tags to metadata for fast retrieval and classification. Users can add detailed annotations to metadata to explain its purpose, source, or association with other data. According to the nature, purpose, or business logic of the data, users can classify metadata for better organization and management. According to factors such as the importance, accuracy, or credibility of the data, users can rate metadata to prioritize data during use.

[0035] In terms of data directory management, the platform provides a wealth of operational functions. Users can create new data directories to organize and manage metadata according to actual needs. Update and adjust existing data directories, including modifying directory names, adjusting classification structures, and updating metadata. For data directories that are no longer needed, users can delete them to release system resources. In addition, the platform also supports the binding and unbinding of data directories and API interfaces. Users can bind specific data directories to API interfaces to provide data access services through the API interfaces. When the content of the data directory changes or the API interface needs to be updated, users can unbind and rebind at any time to ensure the accuracy and stability of data access services. When users need to operate a data source, the platform provides an intuitive operation interface and rich functional options. Users only need to select the target data source and the data operation to be performed (such as query, add, modify, delete, etc.), and the platform can automatically generate and execute the corresponding SQL statements or data access codes according to the user's instructions, thereby realizing flexible operation and management of data sources.

[0036] S106: Call the large model processing module to generate SQL statements suitable for the target database according to the selected target data source and the data operation to be performed, as well as the vector representations of the queries and operation statements in different databases stored in the vector library.

[0037] Call the large model processing module to generate SQL statements suitable for the target database based on the selected target data source and the data operations to be performed, as well as the vector representations of queries and operation statements in different databases stored in the vector library, including: when the user needs to operate on a certain data source, select the target data source and the data operations to be performed. Call the large model processing module, select the target data source and the data operations to be performed, and search the vector representations of queries and operation statements related to the target database operations in the vector library. Search the vector representations of queries and operation statements related to the target database operations in the vector library, select the target data source and the data operations to be performed and input them into the large model to generate SQL statements suitable for the target database. The large model performs standardized answer processing on the generated SQL statements so that the returned data format meets the user's requirements.

[0038] Specifically, in order to achieve flexible data operations across database types, a database operation processing flow based on a large model (taking the GLM large model as an example) is designed. The process covers the complete process from server deployment, vector library construction, data source configuration to SQL statement generation and execution. The following is a detailed step description. Install the GLM large model on a server. The GLM large model is selected as the core tool of this application because of its powerful reasoning performance, long context processing capabilities, multi-language, multi-modality and All Tools. Using the interface provided by the GLM large model, users can easily perform data operations and queries. For different types of databases (such as MYSQL, ArgoDB, etc.), common queries and operation statements are written into TXT texts. These texts are vectorized and stored in the vector library. The existence of the vector library enables the large model to refer to and draw on these vectorized statements when generating SQL statements, thereby improving the accuracy and applicability of the generated statements. For some database types with less information, the adaptability of the large model can be enhanced by continuously updating the TXT text and vector library. The user configures the data source through the platform, adds the data source to the data source list, and binds the corresponding database driver. During the configuration process, the user needs to enter key information such as the name, type, URL, user name, password, and status of the data source. The user selects the database to be queried or operated from the data source list. According to actual needs, the user selects the corresponding data operation type (such as query, insert, update, or delete). Based on the database type and the operation to be performed, the relevant SQL example sentences are searched in the vector library. The SQL example sentences found, the SQL statement parameters entered by the user, and the type of SQL statement required to be generated are input into the GLM large model. The large model generates SQL statements suitable for the target database based on this information. The SQL statements generated by the large model need to be formatted to ensure that they meet the return data format required by the user. The formatted SQL statements are returned to the user in JSON format, including information such as database type, SQL statement, and operation results. According to the SQL statement returned by the large model, the database operation statement is executed to query the database. The query results are returned to the user in the form of a data list. If the execution of the SQL statement fails, the SQL example sentence is no longer directly entered, but the large model is called again to generate a new SQL statement. If the generated SQL statement still fails after executing it again, the user is prompted to update the TXT database syntax file and update the vector library.

[0039] For certain specific types of databases, especially domestic databases or databases with less online information, the large model is often unable to directly generate correct SQL statements due to the lack of necessary database dialect knowledge. In this case, the traditional approach is to conduct targeted training on the large model, but as the number of database types increases, this training becomes more frequent and time-consuming. In order to solve the above problems, the large model processing module of the present application is responsible for automatically generating SQL statements suitable for the target database based on the database operation requirements input by the user. The core idea of ​​this method is to vectorize common queries and operation statements of different databases and store them in a vector library. When a user needs to generate an SQL statement for a specific database, the large model processing module will first search the vector library for the corresponding SQL statement syntax instances, and then input these instances into the large model as a knowledge base, thereby assisting the large model in generating SQL statements that conform to the dialect of the target database.

[0040] The application also includes: optimizing the SQL statements generated by the large model in the large model processing module. The optimization includes index usage optimization, query condition optimization and connection operation optimization.

[0041] Indexes are key structures used in databases to speed up data retrieval. In SQL statements generated by large models, optimizing the use of indexes can significantly improve query speed. For fields frequently used in queries, ensure that they are covered by indexes. This helps reduce full table scans and improve query efficiency. When writing SQL statements, be careful to avoid some operations that cause index failures, such as using functions or expressions to calculate index fields, and starting with wildcards in fuzzy queries. These operations will cause the database to be unable to effectively use indexes, thereby reducing query performance. As data is updated and deleted, indexes may become fragmented, affecting query performance. Therefore, indexes need to be rebuilt and maintained regularly to ensure that they are always in the best condition. Query conditions are the key part of SQL statements for filtering data. Optimizing query conditions can further reduce unnecessary data scanning and processing and improve query efficiency. Whenever possible, try to use exact matches instead of range queries, because exact matches can usually locate target data faster. For indexes on long string fields, consider indexing only the prefix part. This can reduce the storage space and maintenance cost of the index while ensuring query efficiency. Join operations are the key part of SQL statements for merging data from multiple tables. Optimizing join operations can further reduce the time for data scanning and processing and improve query efficiency. Choose the appropriate join type based on query requirements and data volume. Different join types may have significant differences in performance. Make sure that the fields in the join conditions are covered by indexes to reduce the time of join operations. At the same time, avoid using complex expressions or functions in join conditions to avoid invalidating indexes. Whenever possible, minimize the number of join tables. Too many join tables will increase query complexity and resource consumption. If you must join multiple tables, consider using temporary tables or subqueries to simplify the query structure. Through efforts in multiple aspects such as index usage optimization, query condition optimization, and join operation optimization, the efficiency and performance of database operations can be significantly improved, providing users with more efficient and stable data services.

[0042] The application also includes: performing syntax and semantic analysis on the generated SQL statements in the large model processing module to generate SQL statement interpretation information. Based on the characteristics of the SQL statements, an execution plan for the SQL operation statements is generated, and each step of the execution plan and the estimated execution time are displayed to help users understand the execution process of the SQL statements.

[0043] Specifically, in the large model processing module, the SQL statements generated by the large model are strictly checked for syntax to ensure that the statements comply with the SQL language specifications. Subsequently, in-depth semantic analysis is performed to parse the various elements in the statement, such as table names, field names, operators, functions, etc., and the relationships between them, to ensure that the statement is logically correct and meaningful. Based on the results of syntax and semantic analysis, a detailed SQL statement explanation information will be generated. This information will clearly show the structure of the SQL statement, the meaning of each part, and how they work together to achieve specific data operations. For complex SQL statements, the explanation information will also include necessary comments and instructions to help users better understand the intent and logic of the statement. Based on the syntax and semantic analysis, an optimized execution plan will be further generated according to the characteristics of the SQL statement and the structure of the database. The execution plan is a set of detailed steps and instructions for actually executing the SQL statement. The execution plan will list in detail the operation type of each step, such as full table scan, index search, join operation, etc., the data tables and fields involved, the amount of data operated, and the estimated execution time and other key information. This information is presented to the user in an intuitive way, such as a tree diagram, a flowchart or a table, to help users clearly understand the execution process and performance bottlenecks of the SQL statement. By providing SQL statement explanation information and execution plans, this application greatly enhances the user's understanding of the SQL statement execution process. Users can evaluate the performance of SQL statements based on this information, identify potential optimization points, and make corresponding adjustments. In addition, the estimated execution time in the execution plan can also help users better plan data operation tasks and ensure the timeliness and efficiency of data processing.

[0044] The large model processing module includes a vector library management unit. The vector library management unit includes a vector generation subunit and a vector storage subunit. In the vector library management unit, queries and operation statements in different databases are stored in the vector library as a reference resource for the large model when processing database operations. In the vector generation subunit, queries and operation statements in different databases are vectorized to generate vector representations. The generated vector representations are stored in the vector storage subunit as a reference resource for the large model when processing database operations.

[0045] S107: View and manage the API interface list and manage data items through the API interface management module, and implement data operations through the API interface.

[0046] Specifically, through this API interface management module, users can easily set and modify various properties of the API interface, including but not limited to key information such as interface name, access URL, data item list, current status, transaction processing rules, permission allocation, and verification security code. In addition, the API interface management module also provides a detailed data item list management function. Users can make in-depth settings and modifications to data items here, including the name of the data item, node name, associated data source name, SQL statement for performing data operations, necessary setting parameters, and the status of the data item. These configurations provide users with great flexibility, allowing users to customize data items according to their actual needs. The API interface management module supports users to directly perform various operations on the data in the data source list through the data item list, such as query, add, modify, and delete.

[0047] When a user submits a data operation request through the API interface management module, the request is received and parsed, and the SQL statement is extracted. The extracted SQL statement is judged for legality according to the preset data operation rules and security policies. If the judgment result is legal, the SQL statement is allowed to continue to execute. If the judgment result is illegal, the execution of the SQL statement is blocked, and an error message is returned to the user, prompting the user to modify or resubmit the data operation request.

[0048] Specifically, the user can view the list of currently available API interfaces through the API interface management module. The user is allowed to configure, enable or disable the API interface to ensure the security and availability of the API interface. When a user submits a data operation request through an API interface, the request is first received by the API interface management module. The API interface management module parses the received request and extracts the SQL statement that the user wants to execute from the parsed data operation request. According to the preset data operation rules and security policies, the extracted SQL statement is judged for legality. The legality judgment may include checking the syntax correctness of the SQL statement, whether it contains potential security risks, whether it violates database access rights, etc. If the judgment result is legal, that is, the SQL statement complies with all data operation rules and security policies, the SQL statement is allowed to continue to execute to complete the data operation requested by the user. If the judgment result is illegal, that is, the SQL statement has syntax errors, security risks or violations of permissions, the execution of the SQL statement is blocked. For blocked SQL statements, the API interface management module will return an error message to the user. The error message should contain sufficient detailed information so that the user can understand the reason why the SQL statement is blocked and modify or resubmit the data operation request according to the prompt. User feedback and modification: Users can modify SQL statements or resubmit data operation requests based on the returned error information. The modified request will go through the above process again for legality judgment and execution.

[0049] S108: According to the user's request, the database module is called to execute the SQL statement, complete the data operation, and return the operation result.

[0050] S109: The network configuration module provides the configuration function of the server's domain name or IP address, supports the front-end page to be accessed through the specified domain name or IP address, and supports the security code verification mechanism to ensure that the front-end page can be authenticated when accessing the database.

[0051] Specifically, the security code verification mechanism can ensure the security of backend services.

[0052] S110: The service migration module supports migration and expansion between different servers or database systems.

[0053] Specifically, the service migration module is used to migrate and expand services between different servers or database systems. Server migration is to migrate a service from one physical server or virtual machine to another. Database system migration is to migrate the database that the service depends on from one database system to another, which can be from one database type to another, or from one database instance to another, such as from a development environment to a production environment. Expansion is to add new instances or nodes based on existing services to improve the system's processing power, availability or fault tolerance.

[0054] S111: The database module stores and manages the data of the platform, including data source information, data directory information and user information.

[0055] Specifically, data source information includes detailed information of all data sources connected to the platform, such as databases, APIs, file systems, etc., such as connection strings, authentication information, data formats, etc. The data directory is a way to organize and classify data within the platform. It records the source, type, structure, storage location, and other information of the data. Data directory information enables users to easily find, understand, and use data within the platform. User information includes personal information, permission information, login credentials, etc. of platform users. User information is the basis for the platform to implement user management, permission control, and identity authentication.

[0056] The application also includes: providing a user rights management function to divide users into roles and assign rights. Users with different roles have different operation rights and data access rights. When a user logs into the platform, the menus and buttons of the user interface are dynamically generated according to the user's role and permission information.

[0057] Specifically, permission allocation is the process of assigning specific permissions to corresponding users. Permissions determine the operations that users can perform in the system, such as creating, reading, updating or deleting data. Based on the user's role and permission information, the menus and buttons of the user interface are dynamically generated to ensure the security of the system and the protection of data.

[0058] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0059] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0060] The methods, devices or modules described in this application can be implemented in a computer-readable program code mode. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor, a logic gate, a switch, an application-specific integrated circuit (English: Application Specific Integrated Circuit, referred to as: ASIC), a programmable logic controller and an embedded microcontroller. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a pure computer-readable program code mode, the controller can be completely implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0061] like Figure 3 As shown, an embodiment of the present application also provides a multi-source data fusion sharing server based on a large model, including a memory 301 and a processor 302; the memory 301 is used to store computer executable instructions; the processor 302 is used to execute computer executable instructions to implement a sharing method of a multi-source data fusion sharing platform based on a large model described above in an embodiment of the present application.

[0062] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the sharing method of a multi-source data fusion sharing platform based on a large model described above in the embodiment of the present application.

[0063] It can be known from the description of the above implementation mode that a person skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0064] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0065] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A multi-source data fusion and sharing platform based on a large model, characterized by: It includes data source configuration module, data directory management module, large model processing module, API interface management module, network configuration module, service migration module and database module; The data source configuration module is configured to add and configure various types of data sources, enter the name, type, URL, user name, password and status of the data source, add the data source to the data source list, and bind the corresponding database driver; The data directory management module is configured to create, modify and search data directories. The data directory includes a main directory and subdirectories. It supports searching the main directory or subdirectories by keywords. When a main directory is selected, all tables under the main directory are displayed. The large model processing module is configured to receive the database operation requirements input by the user, generate SQL statements applicable to the target database by calling the pre-trained large model interface and according to the vector representations of the query and operation statements in different databases stored in the vector library; The API interface management module is configured to provide an API interface and manage data items so that users can interact with the platform by calling the API interface; The network configuration module is configured to configure the network connection parameters of the platform to ensure normal communication between the platform and the external system; The service migration module is configured to support migration and expansion of platform services; The database module is configured to store and manage the platform's data, including data source information, data directory information, and user information.

2. The multi-source data fusion and sharing platform based on a large model according to claim 1 is characterized in that: The large model processing module includes a vector library management unit; The vector library management unit includes a vector generation subunit and a vector storage subunit.

3. A multi-source data fusion and sharing method based on a large model, characterized in that: The multi-source data fusion sharing platform based on a large model described in any one of claims 1 to 2 comprises: Through the data source configuration module, add and configure various types of data sources, enter the name, type, URL, user name, password and status of the data source so that the platform can connect to the corresponding database system; Add the configured data source to the data source list and bind the corresponding database driver; Using the data directory management module, metadata is collected from the data source, and the collected metadata is divided into data directories; Supports creation, modification, and deletion of data directories, as well as binding and unbinding of data directories and API interfaces; When a user needs to operate a data source, he selects the target data source and the data operation to be performed; data operations include querying, adding, modifying and deleting data; Call the large model processing module to generate SQL statements applicable to the target database based on the selected target data source and the data operation to be performed, as well as the vector representations of the queries and operation statements in different databases stored in the vector library; Through the API interface management module, you can view and manage the API interface list and manage data items, and implement data operations through the API interface; According to user requests, call the database module to execute SQL statements, complete data operations, and return operation results; The network configuration module provides the configuration function of the server's domain name or IP address, supports the front-end page to be accessed through the specified domain name or IP address, and supports the security code verification mechanism to ensure that the front-end page can be authenticated when accessing the database; The service migration module supports migration and expansion between different servers or database systems; The database module stores and manages the platform's data, including data source information, data directory information, and user information.

4. The multi-source data fusion and sharing method based on a large model according to claim 3 is characterized in that: The calling of the large model processing module generates an SQL statement applicable to the target database according to the selected target data source and the data operation to be performed, as well as the vector representations of the query and operation statements in different databases stored in the vector library, including: When the user needs to operate a data source, select the target data source and the data operation to be performed; Call the large model processing module, select the target data source and the data operation to be performed, and search the vector library for the vector representation of the query and operation statements related to the target database operation; Search the vector library for queries and operation statements related to the target database operation, select the target data source and the data operation to be performed and input them into the big model to generate SQL statements suitable for the target database; The big model performs standardized response processing on the generated SQL statements to ensure that the returned data format meets the user's requirements.

5. The multi-source data fusion and sharing method based on a large model according to claim 3 is characterized in that: In the vector library management unit, the queries and operation statements in different databases are stored in the vector library to serve as a reference resource for the large model when processing database operations; In the vector generation subunit, the query and operation statements in different databases are vectorized to generate vector representations; The generated vector representation is stored in the vector storage subunit to serve as a reference resource for the large model when processing database operations.

6. The multi-source data fusion and sharing method based on a large model according to claim 3 is characterized in that: Also includes: In the large model processing module, the SQL statements generated by the large model are optimized; The optimization process includes index usage optimization, query condition optimization and connection operation optimization.

7. The multi-source data fusion and sharing method based on a large model according to claim 6 is characterized in that: Also includes: In the large model processing module, the generated SQL statements are analyzed in terms of syntax and semantics to generate SQL statement explanation information; Generate an execution plan for the SQL statement based on the characteristics of the SQL statement, and display each step of the execution plan and the estimated execution time to help users understand the execution process of the SQL statement.

8. The multi-source data fusion and sharing method based on a large model according to claim 3 is characterized in that: Also includes: Provide user authority management function to divide users into roles and assign authority; Users with different roles have different operation permissions and data access permissions; When a user logs in to the platform, the menus and buttons of the user interface are dynamically generated based on the user's role and permission information.

9. The multi-source data fusion and sharing method based on a large model according to claim 3 is characterized in that: When a user submits a data operation request through the API interface management module, the request is received and parsed, and the SQL statement is extracted; According to the preset data operation rules and security policies, the legality of the extracted SQL statements is judged; If the judgment result is legal, the SQL statement is allowed to continue executing; If the judgment result is illegal, the execution of the SQL statement is blocked and an error message is returned to the user, prompting the user to modify or resubmit the data operation request.

Images