Multi-data-source data management method and device and electronic equipment
Through the data management platform, the database connection is directly used to use data source information and transform SQL statements, which solves the problems of operation complexity and query efficiency in multi-data source management, and realizes efficient multi-data source query and operation.
Patent Information
- Application Number
- CN202411600562.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art requires users to clarify the data source format and access it with the help of specific tools in multi-data source management, which increases the problems of operational complexity and inefficient query efficiency.
Through the data management platform, it directly uses data source information to build database connections, which are converted into SQL statements for different database types, realizing unified access and operation of multiple data sources, reducing system coupling and development costs.
It improves the efficiency of multi-data source query, simplifies operational processes, reduces system complexity and development costs, and enhances the scalability and scalability of the system.
Smart Images

Figure CN119088869A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data management, and in particular to a method, device and electronic device for managing data from multiple data sources. Background Art
[0002] In a data-driven environment, data source management plays the role of a core module, which is responsible for a series of key operations such as data source connection, query and screening. However, facing the management challenges of multiple data sources, the current technical framework requires users to first clarify the format of the data source and use specific tools to access it, which undoubtedly increases the complexity of the operation and affects the query efficiency. In addition, query personnel also need to have a deep understanding of various data sources and their storage locations, which reduces the query efficiency.
[0003] In summary, the query efficiency of database queries with multiple data sources needs to be improved. Summary of the invention
[0004] In view of this, an object of the present invention is to provide a method, device and electronic device for managing data from multiple data sources to improve query efficiency.
[0005] In a first aspect, an embodiment of the present invention provides a method for managing data from multiple data sources, characterized in that the method is applied to a data management platform, the data management platform is used to manage multiple databases, and the data management platform stores data source information of the multiple databases; selecting a first target database; In response to the data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data of the first target database; The data source information of the first target database exists in the data source information of the multiple databases.
[0006] In the embodiments provided in the present application, the corresponding database can be accessed and corresponding operations can be performed on the database using the data source information. The connection with the database can be directly established using the data source information, and the database access can be completed without the aid of specific tools, thereby improving query efficiency.
[0007] One possible approach is that the database type of each of the multiple databases complies with a relational database, the data management platform is used to manage multiple databases of multiple relational databases, and one database type corresponds to one SQL language format; The step of accessing the first target database based on the data source information of the first target database in response to the data management instruction and performing corresponding operations on the data of the first target database comprises: In response to the data management instruction, converting the data management condition information into an SQL statement for the data management condition based on the syntax attribute corresponding to the first target database type; Accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the first target database based on the SQL statement for the data management condition; The data management condition information exists in the data management instruction.
[0008] In one possible manner, the method further comprises: Testing whether the first target database is accessible based on the data source information of the first target database; If accessible, executing the step of accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the first target database based on the SQL statement for the data management condition; If it is inaccessible, an inaccessible prompt message is generated.
[0009] One possible approach is that, in the step of testing whether the first target database is accessed successfully, If the first target database type conforms to the preset processing database type, and the address corresponding to the data source information of the first target database is successfully linked, then the first target database is accessible.
[0010] In one possible manner, the data management instruction includes: a data sorting instruction, and the data management condition information includes: screening and sorting information; and / or The data management instruction includes: a data aggregation instruction, and the data management condition information includes: data aggregation condition information; and / or The data management instruction includes: a data query instruction, and the data management condition information includes: data query condition information; and / or The data management instruction includes: a data deduplication instruction, and the data management condition information corresponds to the data deduplication instruction.
[0011] One possible way is that the data management condition information is a json statement. The SQL statement for the data management condition is: an SQL statement generated based on a combination of grammatical attributes corresponding to the first target database type and a json statement corresponding to the data management condition information.
[0012] One possible approach is that the data management platform has a defined SQL generation interface; The SQL statement for the data management condition is generated in the following manner: Based on the first target database type, the SQL generation interface calls the target to generate SQL statement behavior; Based on the target SQL statement generation behavior, based on the data management condition information and the syntax attributes corresponding to the first target database type, the SQL statement for the data management condition is generated.
[0013] In one possible manner, the method further includes: setting a behavior of generating SQL statements corresponding to each database type.
[0014] In one possible manner, the method further comprises: setting data source information of each database among the multiple databases; The data source information of each database is a meta-mapping constructed based on the database name, database access address, and database type.
[0015] In one possible manner, before the step of selecting the first target database, the method further includes: Selecting a second target database type and determining the second target database; The second target database is at least a part of all databases that conform to the type of the second target database, and the first target database exists in the second target database.
[0016] One possible approach is that the data management platform has a defined connector interface; The step of selecting the type of the second target database and determining the second target database comprises: Based on the second target database type, the connector interface calls a target connector; determining a second target database based on the target connector; The target connector is constructed based on the second target database type and a database corresponding to the second target database type.
[0017] In a second aspect, an embodiment of the present application provides a management device for data from multiple data sources, which is applied to a data management platform, wherein the data management platform stores data source information of multiple databases; Selection module: used to select the first target database; A response module: configured to respond to a data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data of the first target database; The data source information of the first target database exists in the data source information of the multiple databases, the data source information of the first target database exists in the data source information of the multiple databases, the database type of each database in the multiple databases conforms to the relational database, the data management platform is used to manage multiple databases of the multiple relational databases, and one database type corresponds to one SQL language format; The response module is specifically used for: In response to the data management instruction, converting the data management condition information into an SQL statement for the data management condition based on the syntax attribute corresponding to the first target database type; Accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the first target database based on the SQL statement for the data management condition; The data management condition information exists in the data management instruction.
[0018] In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor; and at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method as described in the first aspect by calling the program instructions.
[0019] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.
[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 A flow chart of a method for managing data from multiple data sources provided in an embodiment of the present application; Figure 2A diagram of a management server interface for multi-data source data shown in an exemplary embodiment provided by the present application; Figure 3 A flowchart of step S102 in a method for managing data from multiple data sources provided in an embodiment of the present application; Figure 4 A flowchart of a method for managing data from multiple data sources provided as an example in an embodiment of the present application; Figure 5 A flowchart of the database connection steps in a method for managing data from multiple data sources provided in an embodiment of the present application; Figure 6 A schematic diagram of a query interface in a method for managing data from multiple data sources provided in an embodiment of the present application; Figure 7 The embodiments of the present application provide Figure 3 Flow chart of step S102a; Figure 8 A query class UML class diagram showing an exemplary embodiment provided by the present application; Fig. 9 A query behavior diagram provided for an embodiment of the present application; Fig.10 A flowchart of another method for managing data from multiple data sources provided in an embodiment of the present application; Fig.11 A diagram of another management server interface for multi-data source data shown in an exemplary embodiment provided by the present application; Fig.12 A structural diagram of a multi-data source data management device provided in an embodiment of the present application; Fig.13 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] At present, in a data-driven environment, data source management plays the role of a core module, which is responsible for implementing a series of key operations such as data source connection, query and screening. However, in the face of the management challenges of multiple data sources, the current technical framework requires users to first clarify the format of the data source and use specific tools to access it, which undoubtedly increases the complexity of the operation and affects the query efficiency. In addition, the query personnel also need to have a deep understanding of various data sources and their storage locations, which reduces the query efficiency. Based on this, a multi-data source data management method, device and electronic device provided in an embodiment of the present invention can improve the query efficiency of data.
[0025] First, the terms involved in this application are explained: Relational database: A database that stores and organizes data based on a relational model (i.e., the logical relationship between data). It uses tables to store data, and each table consists of rows and columns, where rows represent records and columns represent fields. The core concepts of relational databases include data tables, primary keys, foreign keys, and relationships between tables. Common types are as follows: MySQL: An open source relational database management system widely used in Web development.
[0026] PostgreSQL: A powerful open source database that supports complex queries and data types.
[0027] SQL Server: A relational database developed by Microsoft that is tightly integrated with Microsoft products.
[0028] Database 2: Abbreviated as DB2, a relational database management system.
[0029] Secondly, to facilitate understanding of this embodiment, the method for managing data from multiple data sources disclosed in the embodiment of the present invention is first introduced in detail.
[0030] Reference Figure 1 The embodiment of the present application provides a method for managing data from multiple data sources, which is applied to a data management platform. The data management platform is used to manage multiple databases. The data management platform stores data source information of multiple databases. The method includes: S101: Select a first target database.
[0031] Here, the first target database corresponds to the database to be operated, for example, Figure 2 The data source name is displayed on the display device of the server, and the user can use an external data device (such as a mouse or keyboard) to select the first target database using the name of the data source.
[0032] It should also be noted that the multiple databases here correspond to the databases that the data management platform expects to manage. Specifically, assuming that the data management platform stores the data source information of databases A, B, C and D, then A, B, C and D correspond to the aforementioned multiple databases.
[0033] S102: In response to the data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data of the first target database.
[0034] Specifically, in the embodiments provided in the present application, the data source information includes a URI (Uniform Resource Identifier) for accessing the data source, such as: a naming mechanism for accessing resources, a host name for storing resources, and a path indicating the name of the resource itself ( Figure 2 not shown).
[0035] It should be noted that, in the embodiments provided in the application, the operations that can be performed on the database are at least one of data query, data sorting, data aggregation and data editing.
[0036] In the prior art, even if the data source information is available, additional database access software, such as Oracle, is generally still required to access the database through information such as the address port, etc. However, through the above method, the corresponding database can be accessed using the data source information, and corresponding operations can be performed on the database. The connection with the database can be directly established using the data source information, and the database access can be completed without the help of specific tools, thereby improving the query efficiency.
[0037] In some situations, the database object managed by the data management platform is a relational database. In this situation, among multiple databases, the database type of each database complies with the relational database.
[0038] Further, each database complies with one of MySQL, PostgreSQL, Microsoft SQLServer and IBM Database 2.
[0039] For example, assume that the databases managed by the data management platform include A, B, C, and D, where the database types of A, B, C, and D are one of MySQL, PostgreSQL, Microsoft SQL Server, and IBM Database 2.
[0040] In this scenario, if you want to query data for different types of databases, one possible way to do it is to write specific SQL query statements or use specific data operation tools for each data source, which is not friendly to non-technical personnel or users who are not good at SQL writing. At the same time, if you want to manage multiple data platforms, you need special interfaces or codes to handle different types of data, which leads to high coupling. As the types of data sources increase, the existing system is difficult to expand effectively. The query and screening methods of different data sources vary greatly and need to be processed separately, which increases the complexity of the system and development costs.
[0041] Reference Figure 3 In order to solve the above problem, the aforementioned step S102: in response to the data management instruction, accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the data of the first target database includes: S102a: In response to the data management instruction, convert the data management condition information into an SQL statement for the data management condition based on the syntax attribute corresponding to the first target database type.
[0042] Here, the first target database type is a database type corresponding to the first target database.
[0043] Different database types have different SQL syntaxes, that is, different syntax attributes. For example, to query table names, for MySQL databases, the SQL syntax is SELECT TABLE NAME FROM INFORMATION SCHEMA.TABLES WHERE TABLE SCHEMA = 'mydatabase,' while for DB2 databases, the syntax for querying table names is SELECT TABNAME FROM SYSCAT.TABLES WHERET ABSCHEMA = 'mydatabase'.
[0044] In this step, the data management condition information in the data management instruction is converted based on the grammatical attributes corresponding to the first target database type, and converted into an SQL statement that conforms to the grammatical attributes corresponding to the first target database type. The SQL statement inherits the grammatical attributes of the database and includes the data management condition information. The SQL statement can be used to operate on the corresponding type of database.
[0045] Specifically, the data management condition information is a json statement, and the SQL statement for the data management condition is: an SQL statement generated based on a combination of a grammatical attribute corresponding to the first target database type and a json statement corresponding to the data management condition information.
[0046] It should be noted that the data management condition information is the data management conditions input by the user. For instance, assuming filtering is taken as an example, the user needs to input data filtering conditions into the data management platform. The data filtering conditions input by the user are regarded as data management condition information. When the user sets the data filtering conditions and clicks the OK button, it is regarded as issuing a data management instruction. It can be seen that if a data management instruction is to be issued, the data management condition information needs to be input in advance. Furthermore, the data management condition information exists in the data management instruction.
[0047] S102b: Access the first target database based on the data source information of the first target database, and perform corresponding operations on the first target database based on the SQL statement for the data management condition.
[0048] When the SQL statement for the data management condition is generated, the management method for the first target database can be determined, and the first target database can be operated based on the management method for the first target database.
[0049] In this embodiment, compared with the prior art of using special interfaces or codes to process different types of data, the data management condition information is converted based on the grammatical attributes corresponding to the database type, so that the generated SQL statement is only responsible for the specific management affairs of the corresponding database type, that is, the generated code only processes the corresponding type of data, which reduces the coupling degree of the system, thereby increasing the scalability of the system and reducing the system complexity and development costs.
[0050] On the basis of the aforementioned embodiments, in some embodiments, in order to determine whether the corresponding operation can be performed on the first target database, it is necessary to perform a link detection on the first target database. In this embodiment, it is necessary to test whether the first target database is successfully accessed based on the data source information of the first target database. If it is accessible, execute the aforementioned S102b: access the first target database based on the data source information of the first target database, and perform the corresponding operation on the first target database based on the SQL statement for the data management condition.
[0051] If the access is unsuccessful, a link failure message is generated.
[0052] One possible design is that if the first target database type meets the preset processing database type, and the address corresponding to the data source information of the first target database is successfully linked, if successful, the first target database is accessible.
[0053] Specifically, the data source information includes: IP address, port, database name, and database type. If you want to access the first target database, you need to determine whether the data source information of the first target database is correct.
[0054] Specifically, first determine the database name of the first target database based on its data source information, determine whether the first target database has a preset processing database type, and link based on the IP address and port of the first target database to determine whether the first target database is successfully linked.
[0055] Specifically, the preset processing database type is the management type of the operation that can be performed by the corresponding data management platform. Exemplarily, the data management platform can specifically manage MySQL, PostgreSQL SQL Server and DB2, then MySQL, PostgreSQL SQL Server and DB2 are regarded as preset processing database types. If the data type corresponding to the first target database is MySQL, or the data type corresponding to the first target database is PostgreSQL, or the data type corresponding to the first target database is SQL Server, or the data type corresponding to the first target database is DB2, then the first target database type is deemed to comply with the preset processing database type, and at the same time, the first target database is linked to the address corresponding to the data source information of the first target database, and it is determined whether the first target database is successfully linked.
[0056] For example, refer to Figure 4 First, determine whether the first target database type conforms to the preset processing database type. If so, determine whether the address corresponding to the data source information of the first target database can be successfully linked. If so, try to establish a link with the first target database according to the data source information. If successfully linked, the first target database is accessible.
[0057] The following will describe the data management condition information and data management instructions provided in the embodiments of the present application: Reference Figure 5 In some situations, the user needs to query the data of a certain database (i.e., the first target database mentioned above). At this time, the user enters the query conditions. More specifically, the user can use =, >=, <=, >, <, !=, include, not include, and between to set the query conditions.
[0058] Assuming that multiple query conditions need to be set, the logical relationship between the query conditions can be expressed using Boolean operators.
[0059] For example, refer to Figure 5 When the user enters query condition A, he clicks Add Query Condition. Then, a dialog box for query condition input is added to the server display interface, and the user can enter the added query condition.
[0060] In this context, the data management instruction corresponds to the data query instruction, and the data management condition information corresponds to the data query condition information.
[0061] In another situation, data needs to be aggregated. Here, the user needs to input aggregation conditions. At this time, the data management instruction corresponds to the data aggregation instruction, and the data management condition information corresponds to the data aggregation condition information.
[0062] Specifically, aggregation is to perform operations such as sum, average, maximum, minimum, and total (total number of rows) on a column.
[0063] For example, suppose that the user needs to select a column in Table B in database A to take the average value. In this example, after the user selects Table B in database A, enters the selected column and selects to take the average, and clicks the aggregation operation button using an external input device, which is regarded as issuing a data aggregation instruction. In this example, the column selected by the user and the average are regarded as data aggregation condition information.
[0064] In another scenario, the user needs to sort the data in the first target database. In this scenario, the data management condition information corresponds to the data sorting condition information, and the data management instruction corresponds to the data sorting instruction.
[0065] For example, assuming that the user wants to sort column A in descending order, the user enters the column name corresponding to column A and selects descending order, and then a data sorting instruction can be issued.
[0066] In another situation, the user needs to deduplicate the data in the database. In this case, the data management instruction corresponds to the data deduplication instruction, and the data management condition information corresponds to the data management instruction.
[0067] For data deduplication, that is, removing duplicate data.
[0068] It can be seen that the data management instruction includes: data query instruction, and the data management condition information includes: data query condition information; and / or The data management instruction includes: a data aggregation instruction, and the data management condition information includes: data aggregation condition information; and / or The data management instruction includes: a data sorting instruction, and the data management condition information includes: data sorting condition information; and / or The data management instruction includes: a data deduplication instruction, and the data management condition information corresponds to the data deduplication instruction.
[0069] The above is a specific description of the data management instructions and data management condition information in the embodiments of the present application. In summary, it can be seen that the above embodiments can specifically perform operations such as data query, sorting, deduplication and aggregation.
[0070] Based on the above embodiments, Figure 6 ,The data management platform assumes that the user needs to query part of the data and save the ,query results. The user can first enter the data ,query condition information and issue the data query command. Then, the server displays the ,query results and saves the current query results to the ,customized data source.
[0071] The following describes the generation method of the aforementioned SQL statements for data management conditions: In combination with the foregoing, it can be seen that the data management platform can manage relational databases of multiple database types, such as MySQL, PostgreSQL, DB2 and SQL Server. In the embodiments provided in this application, for each type of database, the behavior of generating SQL statements is abstracted and encapsulated, and a unified SQL generation interface is set. The SQL generation interface can specifically call the behavior of generating SQL statements for all database types.
[0072] Reference Figure 7 In order to generate the aforementioned SQL statement for the data management condition, first execute S201: based on the first target database type, the SQL generation interface calls the target to generate SQL statement behavior, and then execute S202: based on the target to generate SQL statement behavior, as well as the data management condition information and the syntax attributes corresponding to the first target database type, generate the SQL statement for the data management condition.
[0073] It should be noted that, in the embodiment, the target SQL statement generation behavior is the SQL statement generation behavior corresponding to the first target database type. For example, assuming that the first target database is A, the database type of A database is MySQL, and the data management condition information is input, it is necessary to generate SQL statements for the data management conditions.
[0074] In this example, MySQL corresponds to the first target database type. Based on the mapping of the MySQL database type, the SQL generation interface is called to call the corresponding MySQL SQL statement generation behavior. The corresponding MySQL SQL statement generation behavior is used, and the josn statement corresponding to the data management condition information and the corresponding syntax attributes of MySQL are used to generate SQL statements for the data management conditions.
[0075] Through the above method, each database type is provided with a corresponding database interface, that is, the corresponding SQL statement can be generated according to the given database type, which reduces the coupling of internal codes, simplifies the calling logic of the system and improves the reusability and scalability of the code.
[0076] Furthermore, in the embodiments provided in the present application, specifically, in order to construct and generate SQL statements for data management conditions, the query class corresponding to each database type can be dynamically set.
[0077] For example, continue to take the MySQL query class as an example. For the MySQL query class that inherits the BaseQuery class corresponding to MySQL, refer to Figure 8 .
[0078] In this example, the behavior of generating SQL statements can be regarded as a query behavior of the query class. The BaseQuery class uses the SQL generation interface to call the corresponding MySQL behavior of generating SQL statements, and the query class corresponding to MySQL also inherits the syntax attributes of MySQL in the BaseQuery class. Through the above method, the query class corresponding to MySQL can be constructed. The query class includes the corresponding SQL statement generation method and syntax attributes of MySQL, and generates the corresponding SQL statement according to the given database type, which reduces the coupling of internal codes and improves the scalability of components.
[0079] From the above, we can see that through the unified SQL generation interface, a unified calling method for different database operations is realized. The design simplifies the system's calling logic, reduces the coupling between components in the data management platform, and improves the reusability and scalability of the code.
[0080] Regarding the behavior of generating SQL statements, the behavior of generating SQL statements can be dynamically set. In the embodiment, the behavior of generating SQL statements corresponding to each database type needs to be set.
[0081] Reference Fig. 9 In addition to generating SQL statements, query behaviors also include: obtaining database table name behavior, obtaining column names and connectors of a table. Each query behavior corresponds to a unified interface. By referring to the aforementioned method of building query classes, corresponding query classes can be built for multiple databases. By defining multiple query behaviors, the query class can be expanded, which is conducive to the further maintenance and development of the system.
[0082] Taking obtaining the database table name as an example, the data management platform defines a database table name interface. Specifically, the query class can be constructed in the following way: First, based on the database type, call the behavior of obtaining the database table name corresponding to the database type, that is, obtaining the target database table name behavior. The BaseQuery class corresponding to the database type executes the action of obtaining the database table name based on the target behavior of obtaining the database table name. The query class constructed in the above way inherits the syntax properties in the BaseQuery class and includes the method of obtaining the database table name. You can get the database table name by running the query class.
[0083] On the basis of the above-mentioned embodiment, in order to realize the management of the database, it is necessary to set the data source information of each database among the multiple databases.
[0084] Assume that the databases that the data management platform expects to manage include A, B, C and D. Combined with the foregoing, A, B, C and D correspond to multiple databases. Furthermore, the data source information of A, B, C and D needs to be set.
[0085] Furthermore, a meta-mapping may be constructed using the database name and the database access address. Specifically, the meta-mapping may be constructed using the database name and the database port, the database access address and the database type. In this scenario, the data source information of each database is a meta-mapping constructed based on the database name, the database access address and the database type.
[0086] The database constructed using the above method adopts the mechanism of meta-management and meta-mapping. The data management platform can automatically identify the database type of each data source, and automatically identify and adapt the structure and characteristics of different data sources based on the data type.
[0087] Reference Fig.10 , Fig.10 Another method for managing data from multiple data sources provided in an embodiment of the present application is applied to a data management platform, the data management platform is used to manage multiple databases, and the data management platform stores data source information of multiple databases. The method includes: S301: Select the type of the second target database and determine the second target database.
[0088] Here, the second target database type corresponds to the database selected by the user.
[0089] It should be noted that in some examples, the second target database type conforms to the first target database type. For example, the user expects to query all database names under the MySQL data type. In this case, MySQL corresponds to the second target database type, and all databases under the MySQL type are displayed.
[0090] S302: Select a first target database.
[0091] S303: In response to the data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data in the first target database.
[0092] Reference Fig.12 , assuming that the second target database selected by the user is MySQL, the server will display the data source name, database name, address, etc. of all databases.
[0093] The data source information of the first target database exists in the data source information of multiple databases, the second target database is at least a part of all databases that conform to the target database type, and the first target database exists in the second target database.
[0094] For S302 and S303, please refer to the aforementioned steps S101 and S102 for details, which will not be repeated here.
[0095] It should be noted that the difference between the first target database type and the second target database type depends on the scenario, and the first target database type conforms to the second target database type.
[0096] For example, the databases corresponding to MySQL are A, B, and C. The user needs to perform a query operation on database A. Combined with the above, database A corresponds to the first target database mentioned above, and the database type of database A is MySQL, that is, the first target database type MySQL.
[0097] The beneficial effects of the embodiments of the present application are specifically described above and will not be repeated here.
[0098] As a preferred embodiment, compared with the above, a connector can be used to determine the second target database. Specifically, the connector is a query behavior, and the connector is specifically used to build a mapping between a database type and a database corresponding to the database type.
[0099] In this embodiment, the data management platform has a defined connector interface, and the aforementioned step S301: selecting a target database type and determining a second target database includes: S301a: Based on the second target database type, the connector interface calls the target connector.
[0100] It should be noted that the target connector is a connector corresponding to the second target database type.
[0101] S301b: Determine a second target database based on the target connector.
[0102] The target connector is constructed based on the second target database type and a database corresponding to the second target database type.
[0103] Through the above method, it is possible to ensure that the correct database connector is dynamically selected according to the database type, providing the necessary flexibility and scalability for data query and filtering operations.
[0104] Reference Fig.12 , the embodiment of the present application also provides a management device for multi-data source data, which is applied to a data management platform, and the data management platform stores data source information of multiple databases; Selection module: used to select the first target database; A response module: configured to respond to the data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data of the first target database; The data source information of the first target database exists in the data source information of the multiple databases, the data source information of the first target database exists in the data source information of the multiple databases, the database type of each database in the multiple databases conforms to the relational database, the data management platform is used to manage multiple databases of the multiple relational databases, and one database type corresponds to one SQL language format; The response module is specifically used for: In response to the data management instruction, converting the data management condition information into an SQL statement for the data management condition based on the syntax attribute corresponding to the first target database type; Accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the first target database based on the SQL statement for the data management condition; The data management condition information exists in the data management instruction.
[0105] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0106] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0107] Fig.13 A block diagram of an exemplary electronic device suitable for implementing an exemplary embodiment of the present invention is shown. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0108] like Fig.13 As shown, the electronic device is in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, a memory 430, and a communication bus 440 connecting different system components (including the memory 430 and the processing unit 410).
[0109] The communication bus 440 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnection (PCI) bus.
[0110] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.
[0111] The memory 430 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of each embodiment of the present invention.
[0112] A program / utility having a set (at least one) of program modules may be stored in the memory 430, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described in the embodiments of the present invention.
[0113] The processor 410 executes various functional applications and data processing by running the programs stored in the memory 430, such as implementing the embodiments of the present invention. Figure 1 to Figure 11 The illustrated embodiment provides a method for managing data from multiple data sources.
[0114] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
Claims
1. A method for managing data from multiple data sources, characterized in that: Applied to a data management platform, the data management platform is used to manage multiple databases, and the data management platform stores data source information of the multiple databases; selecting a first target database; In response to the data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data of the first target database; The data source information of the first target database exists in the data source information of the multiple databases, the database type of each database in the multiple databases conforms to the relational database, the data management platform is used to manage multiple databases of the multiple relational databases, and one database type corresponds to one SQL language format; The step of accessing the first target database based on the data source information of the first target database in response to the data management instruction and performing corresponding operations on the data of the first target database comprises: In response to the data management instruction, converting the data management condition information into an SQL statement for the data management condition based on the syntax attribute corresponding to the first target database type; Accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the first target database based on the SQL statement for the data management condition; The data management condition information exists in the data management instruction.
2. The method according to claim 1, characterized in that The method further comprises: testing whether the first target database is accessible based on the data source information of the first target database; If accessible, executing the step of accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the first target database based on the SQL statement for the data management condition; If it is inaccessible, an inaccessible prompt message is generated.
3. The method according to claim 1, characterized in that The data management instructions include: data sorting instructions, and the data management information includes: screening and sorting information; and / or The data management instruction includes: a data aggregation instruction, and the data management condition information includes: data aggregation condition information; and / or The data management instruction includes: a data query instruction, and the data management condition information includes: data query condition information; and / or The data management instruction includes: a data deduplication instruction, and the data management condition information corresponds to the data deduplication instruction.
4. The method according to claim 1, characterized in that The data management condition information is a json statement. The SQL statement for the data management condition is: an SQL statement generated based on a combination of grammatical attributes corresponding to the first target database type and a json statement corresponding to the data management condition information.
5. The method according to claim 4, characterized in that The data management platform has a defined SQL generation interface; The SQL statement for the data management condition is generated in the following manner: Based on the first target database type, the SQL generation interface calls the target to generate SQL statement behavior; Based on the target SQL statement generation behavior, based on the data management condition information and the syntax attributes corresponding to the first target database type, the SQL statement for the data management condition is generated.
6. The method according to claim 5, characterized in that The method further includes: setting a behavior of generating SQL statements corresponding to each database type.
7. The method according to claim 1, characterized in that The method further includes: setting data source information of each database among the plurality of databases; The data source information of each database is a meta-mapping constructed based on the database name, database access address, and database type.
8. A device for managing data from multiple data sources, characterized in that: Applied to a data management platform, the data management platform stores data source information of multiple databases; Selection module: used to select the first target database; A response module: configured to respond to a data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data of the first target database; The data source information of the first target database exists in the data source information of the multiple databases, the database type of each database in the multiple databases conforms to the relational database, the data management platform is used to manage multiple databases of the multiple relational databases, and one database type corresponds to one SQL language format; The response module is specifically used for: In response to the data management instruction, converting the data management condition information into an SQL statement for the data management condition based on the syntax attribute corresponding to the first target database type; Accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the first target database based on the SQL statement for the data management condition; The data management condition information exists in the data management instruction.
9. An electronic device, characterized in that: include: at least one processor; as well as at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 8 by calling the program instructions.
Citation Information
Patent Citations
Data access method and device and electronic equipment
CN108763530A
Multi-source database access method and device, storage medium and electronic equipment
CN109614427A
Database statement execution method and related device
CN110674162A
Data processing device, data processing method, storage medium and electronic equipment
CN111949693A
Program statement conversion method, device, equipment and storage medium
CN114090613A
