Method and device for managing data of multiple data sources

Abstract

The invention provides a multi-data-source data management method and device, and relates to the field of data management.The method is applied to a data management platform, the data management platform is used for managing multiple databases, the data management platform stores data source information of the multiple databases, and the method comprises the steps that a first target database is selected; and in response to a data management instruction, accessing a first target database based on data source information of the first target database, and executing a corresponding operation on data of the first target database. According to the method and the device, the corresponding database can be accessed by utilizing the data source information, the corresponding operation is executed on the database, the connection with the database is constructed by directly utilizing the data source information, the access of the database can be completed without a specific tool, and the query efficiency is improved.

Description

Technical Field

[0001] This invention relates to the technical field of data management, and in particular to a method and apparatus for managing data from multiple data sources. Background Technology

[0002] In a data-driven environment, data source management plays a core role, responsible for key operations such as connecting to, querying, and filtering data sources. However, facing the challenges of managing multiple data sources, current technical frameworks require users to first understand the data source format and access it using specific tools. This undoubtedly increases operational complexity and impacts query efficiency. Furthermore, query personnel need in-depth knowledge of various data sources and their storage locations, which further reduces query efficiency.

[0003] In conclusion, the query efficiency for databases with multiple data sources needs to be improved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and apparatus for managing data from multiple data sources, so as to improve query efficiency.

[0005] In a first aspect, embodiments of the present invention provide a method for managing data from multiple data sources, characterized in that it is applied to a data management platform, the data management platform being used to manage multiple databases, the data management platform storing data source information of multiple databases, and based on a unified data access specification, the unified data access specification defining a set of standard data operations, which are implemented by several data operation executors respectively, each data operation executor being configured to provide specific functional logic for a specific operation in the standard data operations; Select the first target database; In response to a data management instruction, a data operation executor that implements the unified data access specification and corresponds to the data management instruction is invoked; during the execution of the data operation executor, the first target database is accessed based on the data source information of the first target database, and corresponding operations are performed on the data in the first target database; The data source information of the first target database is contained within the data source information of the plurality of databases.

[0006] In the embodiments provided in this application, the corresponding database can be accessed and corresponding operations can be performed on the database using the data source information. The connection with the database can be established directly using the data source information, and database access can be completed without the need for specific tools, thereby improving query efficiency.

[0007] One possible approach is that each of the multiple databases is a relational database, and the data management platform is used to manage multiple databases of various relational databases, with one database type corresponding to one SQL language format; The steps of responding to a data management instruction, accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the data in the first target database include: In response to data management instructions, the data management condition information is converted into SQL statements for the data management conditions based on the syntax attributes corresponding to the first target database type. Access the first target database based on the data source information of the first target database, and perform corresponding operations on the first target database based on the SQL statement for the data management conditions; The data management condition information is contained within the data management instructions.

[0008] One possible approach is that the method further includes: Test whether the first target database is accessible based on the data source information of the first target database; If accessible, the steps of accessing the first target database based on the data source information of the first target database and performing corresponding operations on the first target database based on the SQL statement for data management conditions are executed. If the access is denied, an access denied message will be generated.

[0009] One possible approach is to include the test in the step of determining whether the first target database was successfully accessed. If the first target database type matches the preset processing database type, and the address corresponding to the data source information of the first target database is successfully linked, then the first target database is accessible.

[0010] One possible approach is that the data management instructions include: data sorting instructions, and the data management condition information includes: filtering and sorting information; and / or The data management instructions include: data aggregation instructions, and the data management condition information includes: data aggregation condition information; and / or The data management instructions include: data query instructions, and the data management condition information includes: data query condition information; and / or The data management instructions include: a data deduplication instruction, and the data management condition information corresponds to the data deduplication instruction.

[0011] One possible approach is that the data management condition information is a JSON statement. The SQL statement for the data management conditions is an SQL statement generated based on the syntax attributes corresponding to the first target database type and the JSON statement corresponding to the data management condition information.

[0012] One possible approach is that the data management platform has a defined SQL generation interface; The SQL statement for the data management conditions is generated as follows: Based on the first target database type, the SQL generation interface calls the target to generate SQL statements. Based on the target, the SQL statement is generated, and based on the data management condition information and the syntax attributes corresponding to the first target database type, the SQL statement for the data management conditions is generated.

[0013] One possible approach is that the method also includes setting the SQL statement generation behavior for each database type.

[0014] One possible approach is that the method further includes: the SQL generation interface calling the target to generate SQL statements, which is implemented by calling an algorithm, the calling algorithm including the following steps: S1: Use the type identifier of the first target database as the input parameter; S2: Calculate the hash value of the type identifier, and based on the consistent hashing algorithm, map the hash value to a virtual node ring to determine the nearest virtual node in the clockwise direction along the ring; S3: Determine the corresponding target SQL statement generation behavior based on the preset mapping relationship between virtual nodes and SQL statement generation behavior; S4: Invoke the determined target SQL statement generation behavior to generate an SQL statement that conforms to the syntax of the first target database; When the data management platform is expanded to support new database types, the impact on the mapping relationship between existing database types and SQL statement generation behavior is minimized by adding corresponding virtual nodes and SQL statement generation behavior for the new database types.

[0015] One possible approach is that the invocation of the data operation executor, which corresponds to the data management instruction and implements the unified data access specification, includes: The data management instructions are parsed to determine their operation type; Based on the operation type, from among several data operation executors that implement the unified data access specification, a specific data operation executor configured to handle that operation type is located; Invoke the specific data operation executor located.

[0016] One possible approach is that the execution process of the invoked specific data manipulation executor includes: The data operation executor receives data management condition information from the data management instruction; The data operation executor adapts to and calls the statement generation logic corresponding to the type of the first target database to generate operation statements that conform to the syntax of the first target database. The data operation executor calls the data connector, establishes a connection using the data source information of the first target database, and executes the generated operation statements.

[0017] One possible approach is that the method further includes setting data source information for each of the multiple databases; The data source information for each database is a meta-mapping constructed based on the database name, database access address, and database type.

[0018] One possible approach is that, prior to the step of selecting the first target database, the method further includes: Select the second target database type and determine the second target database; The second target database is at least a portion of all databases that conform to the second target database type, and the first target database exists within the second target database.

[0019] One possible approach is that the data management platform has a defined connector interface; The steps of selecting the second target database type and determining the second target database include: Based on the second target database type, the connector interface calls the target connector; Determine the second target database based on the target connector; The target connector is constructed based on the second target database type and the database corresponding to the second target database type.

[0020] Secondly, embodiments of this application provide a management device for multiple data sources, applied to a data management platform, wherein the data management platform stores data source information of multiple databases; Selected module: Used to select the first target database; Response module: Used to respond to data management instructions, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data in the first target database; The data source information of the first target database is contained within the data source information of the plurality of databases.

[0021] Thirdly, embodiments of this application provide an electronic device, including: At least one processor; and At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the method as described in the first aspect.

[0022] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a method for managing data from multiple data sources, as provided in this application embodiment; Figure 2 An exemplary embodiment provided in this application illustrates a management server interface diagram for multi-source data. Figure 3 A flowchart of step S102 in a method for managing data from multiple data sources provided in this application embodiment; Figure 4 A flowchart illustrating a method for managing data from multiple data sources is provided as an example in this application embodiment; Figure 5 A flowchart of the database connection steps in a method for managing data from multiple data sources provided in this application embodiment; Figure 6This application provides a schematic diagram of a query interface in a method for managing data from multiple data sources. Figure 7 The embodiments provided in this application Figure 3 Flowchart of steps S102a; Figure 8 A query class UML class diagram shown in an exemplary embodiment provided in this application; Figure 9 A query behavior graph provided for embodiments of this application; Figure 10 A flowchart illustrating another method for managing multi-source data provided in this application embodiment; Figure 11 A diagram illustrating another management server interface for multi-data source data provided in an exemplary embodiment of this application. Figure 12 A structural diagram of a multi-source data management device provided in this application embodiment; Figure 13 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Currently, in data-driven environments, data source management plays a core role, responsible for key operations such as connecting to, querying, and filtering data sources. However, facing the challenges of managing multiple data sources, current technical frameworks require users to first understand the data source format and access it using specific tools. This undoubtedly increases operational complexity and impacts query efficiency. Furthermore, query personnel need in-depth knowledge of various data sources and their storage locations, further reducing query efficiency. Therefore, this invention provides a method, apparatus, and electronic device for managing multiple data sources, which can improve data query efficiency.

[0028] First, let me explain the terminology used in this application: Relational databases are databases that store and organize data based on a relational model (i.e., logical relationships between data). They use tables to store data, each table consisting of rows and columns, where rows represent records and columns represent fields. Core concepts of relational databases include data tables, primary keys, foreign keys, and relationships between tables. Common types are as follows: MySQL: An open-source relational database management system widely used in web development.

[0029] PostgreSQL: A powerful open-source database that supports complex queries and data types.

[0030] SQL Server: A relational database developed by Microsoft and tightly integrated with Microsoft products.

[0031] Database 2 (DB2 for short) is a relational database management system.

[0032] Secondly, to facilitate understanding of this embodiment, the method for managing multi-source data disclosed in this embodiment will be described in detail.

[0033] Reference Figure 1 This application provides a method for managing data from multiple data sources, applied to a data management platform. The data management platform manages multiple databases and stores data source information for these databases. The method includes: S101: Select the first target database.

[0034] Here, the first target database corresponds to the database to be operated on, for example, referring to... Figure 2 The server's display device shows the data source name, and users can use external data devices (such as a mouse or keyboard) to select the first target database by the data source name.

[0035] It should also be noted that the multiple databases mentioned here refer to the databases that the data management platform is expected to manage. Specifically, assuming that the data management platform stores the data source information of databases A, B, C, and D, then A, B, C, and D correspond to the aforementioned multiple databases.

[0036] S102: In response to the data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data in the first target database.

[0037] Specifically, in the embodiments provided in this application, the data source information includes a URI (Uniform Resource Identifier) ​​used to access the data source, such as: the naming mechanism for accessing the resource, the hostname where the resource is stored, and the path representing the name of the resource itself (…). Figure 2 (Not shown in the image).

[0038] It should be noted that, in the embodiments provided in this application, the operations that can be performed on the database are at least one of data query, data sorting, data aggregation, and data editing.

[0039] In existing technologies, even with data source information, additional database access software, such as Oracle, is generally required to access the database via address and port information. However, the method described above allows access to the corresponding database and execution of corresponding operations using only the data source information. It directly establishes a connection with the database using the data source information, eliminating the need for specific tools and improving query efficiency.

[0040] In some scenarios, the database objects managed by the data management platform are relational databases. In this scenario, each of the multiple databases is a relational database.

[0041] Furthermore, each database conforms to one of MySQL, PostgreSQL, Microsoft SQL Server, and IBM Database 2.

[0042] For example, suppose the data management platform manages databases A, B, C, and D, where the database type of A, B, C, and D is one of MySQL, PostgreSQL, Microsoft SQL Server, and IBM Database 2.

[0043] In this context, to implement data queries for different types of databases, one possible approach is to write specific SQL queries or use specific data manipulation tools for each data source. This is not user-friendly for non-technical users or those unfamiliar with SQL. Furthermore, managing multiple data platforms requires dedicated interfaces or code to handle different data types, leading to high coupling. As the number of data source types increases, the existing system struggles to scale effectively. The query and filtering methods for different data sources vary significantly, requiring separate handling and increasing system complexity and development costs.

[0044] Reference Figure 3To address the aforementioned issues, the steps in S102—responding to a data management instruction, accessing the first target database based on its data source information, and performing corresponding operations on the data in the first target database—include: S102a: In response to a data management instruction, the data management condition information is converted into an SQL statement for the data management condition based on the syntax attributes corresponding to the first target database type.

[0045] Here, the first target database type is the database type corresponding to the first target database.

[0046] The SQL syntax differs depending on the database type; that is, the syntax attributes also differ. For example, to query a table name, the SQL syntax for a MySQL database is `SELECT TABLE NAME FROM INFORMATION SCHEMA.TABLES WHERE TABLE SCHEMA = 'mydatabase'`, while for a DB2 database, the syntax is `SELECT TABNAME FROM SYSCAT.TABLES WHERE TABLE SCHEMA = 'mydatabase'`.

[0047] In this step, the data management condition information in the data management instruction is converted based on the syntax attributes corresponding to the first target database type, and then converted into an SQL statement that conforms to the syntax attributes corresponding to the first target database type. This SQL statement inherits the syntax attributes of the database and includes the data management condition information. This SQL statement can be used to operate on the corresponding type of database.

[0048] Specifically, the data management condition information is a JSON statement, while the SQL statement for the data management condition is an SQL statement generated by combining the syntax attributes corresponding to the first target database type and the JSON statement corresponding to the data management condition information.

[0049] It should be noted that the data management condition information refers to the data management conditions entered by the user. For example, taking filtering as an example, the user needs to enter data filtering conditions into the data management platform. The data filtering conditions entered by the user are regarded as data management condition information. When the user sets the data filtering conditions and clicks the OK button, it is considered that a data management instruction has been issued. Thus, it can be seen that if a data management instruction is to be issued, the data management condition information needs to be entered in advance. Furthermore, the data management condition information exists in the data management instruction.

[0050] S102b: Access the first target database based on the data source information of the first target database, and perform corresponding operations on the first target database based on SQL statements for data management conditions.

[0051] Once the SQL statement for the data management conditions is generated, the management method for the first target database can be determined, and operations can be performed on the first target database based on the management method for the first target database.

[0052] In this embodiment, compared to the prior art which uses dedicated interfaces or code to process different types of data, the data management condition information is converted based on the syntax attributes corresponding to the database type. This makes the generated SQL statements only responsible for the specific management transactions of the corresponding database type. In other words, the generated code only processes the corresponding type of data, which reduces the coupling of the system, thereby increasing the system's scalability and reducing the system's complexity and development cost.

[0053] Based on the foregoing embodiments, in some embodiments, in order to determine whether the corresponding operation can be performed on the first target database, it is necessary to perform a link detection on the first target database. In this embodiment, it is necessary to test whether the first target database is successfully accessed based on the data source information of the first target database. If it can be accessed, then the aforementioned S102b is executed: accessing the first target database based on the data source information of the first target database, and performing the corresponding operation on the first target database based on the SQL statement for the data management conditions.

[0054] If the access fails, a link failure message will be generated.

[0055] One possible design is that if the type of the first target database matches the preset processing database type, and the address corresponding to the data source information of the first target database is successfully linked, then the first target database can be accessed.

[0056] Specifically, the data source information includes: IP address, port, database name, and database type. To access the first target database, it is necessary to verify that the data source information of the first target database is correct.

[0057] Specifically, the database name is first determined based on the data source information of the first target database, it is determined whether the first target database has a preset processing database type, and a connection is made based on the IP address and port of the first target database to determine whether the connection to the first target database is successful.

[0058] Specifically, the preset processing database type corresponds to the management type that the data management platform can perform operations on. For example, if the data management platform can manage MySQL, PostgreSQL, SQL Server, and DB2, then MySQL, PostgreSQL, SQL Server, and DB2 are considered the preset processing database types. If the data type corresponding to the first target database is MySQL, or PostgreSQL, or SQL Server, or DB2, then the first target database type is considered to conform to the preset processing database type. Simultaneously, a connection is established to the first target database based on the address corresponding to the data source information of the first target database, and it is determined whether the connection to the first target database is successful.

[0059] For example, refer to Figure 4 First, determine whether the type of the first target database matches the preset processing database type. If it does, determine whether the address corresponding to the data source information of the first target database can be successfully connected. If the connection is successful, attempt to establish a connection with the first target database based on the data source information. If the connection is successful, the first target database can be accessed.

[0060] The following will describe the data management conditions and data management instructions provided in the embodiments of this application: Reference Figure 5 In some scenarios, users need to query data in a certain database (i.e., the aforementioned first target database). In this case, the user enters the query conditions. More specifically, the user can use =, >=, <=, >, <, !=, include, exclude, and set the query conditions.

[0061] If you need to set multiple query conditions, you can use Boolean operators to express the logical relationship between each query condition.

[0062] For example, refer to Figure 5 When a user enters query condition A and clicks "Add Query Condition," the server displays a new dialog box for entering query conditions, allowing the user to input the added query conditions.

[0063] In this context, data management commands correspond to data query commands, and data management condition information corresponds to data query condition information.

[0064] In another scenario, data aggregation is required. Here, the user needs to input aggregation conditions. In this case, the data management command corresponds to the data aggregation command, and the data management condition information corresponds to the data aggregation condition information.

[0065] Specifically, aggregation involves performing operations such as summing, averaging, finding the maximum or minimum value, or totaling (total number of rows) on a specific column.

[0066] For example, suppose a user needs to select a column in table B of database A and calculate the average. In this example, after the user selects table B in database A, enters the selected column and selects to calculate the average, clicking the aggregation operation button using an external input device is considered as issuing a data aggregation command. In this example, the column selected by the user and the calculation of the average are considered as data aggregation condition information.

[0067] In another scenario, the user needs to sort the data in the first target database. In this scenario, the data management condition information corresponds to the data sorting condition information, and the data management instruction corresponds to the data sorting instruction.

[0068] For example, suppose a user wants to sort column A in descending order. In this case, the user enters the column name of column A and selects descending order, and the data sorting command can be issued.

[0069] In another scenario, users need to deduplicate data in the database. In this case, the data management command corresponds to the data deduplication command, and the data management condition information corresponds to the data management command.

[0070] This refers to data deduplication, i.e., removing duplicate data.

[0071] Therefore, data management instructions include: data query instructions, and data management condition information includes: data query condition information; and / or Data management instructions include: data aggregation instructions; data management condition information includes: data aggregation condition information. and / or Data management instructions include: data sorting instructions; data management condition information includes: data sorting condition information. and / or The data management instructions include: data deduplication instructions, wherein the data management condition information corresponds to the data deduplication instructions.

[0072] The above is a detailed description of the data management instructions and data management condition information in the embodiments of this application. As can be seen from the above, the embodiments can specifically perform operations such as data query, sorting, deduplication and aggregation.

[0073] Based on the above embodiments, referring to Figure 6 The data management platform assumes that users need to query some data and save the query results. Users can first enter data query conditions and issue data query instructions. Then, the server displays the query results and saves the current query results to a custom data source.

[0074] The following section elaborates on the methods for generating SQL statements based on data management conditions: As described above, the data management platform can manage relational databases of multiple database types, such as MySQL, PostgreSQL, DB2, and SQL Server. In the embodiments provided in this application, the SQL statement generation behavior is abstracted and encapsulated for each type of database, and a unified SQL generation interface is set up. The SQL generation interface can specifically call the SQL statement generation behavior of all database types.

[0075] Reference Figure 7 In order to generate the aforementioned SQL statement for the data management conditions, firstly, S201 is executed: based on the first target database type, the SQL generation interface calls the target to generate SQL statement behavior. Then, S202 is executed: based on the target to generate SQL statement behavior, as well as the data management condition information and the syntax attributes corresponding to the first target database type, the SQL statement for the data management conditions is generated.

[0076] It should be noted that, in the embodiments, the target SQL statement generation behavior is the SQL statement generation behavior corresponding to the first target database type. For example, assuming that the first target database is A, and the database type of database A is MySQL, it is used to input data management condition information and needs to generate SQL statements for the data management conditions.

[0077] In this example, MySQL corresponds to the first target database type. Based on the mapping of the MySQL database type, the SQL generation interface is called to call the corresponding SQL statement generation behavior of MySQL. Using the corresponding SQL statement generation behavior of MySQL, the JOSN statement corresponding to the data management condition information and the corresponding syntax attributes of MySQL, the SQL statement for the data management condition can be generated.

[0078] Furthermore, the process of the SQL generation interface calling the target to generate SQL statements is implemented through an efficient calling algorithm to ensure scheduling accuracy and system scalability. The core steps of this calling algorithm are as follows: Step S1: Input parameter processing. The algorithm uses the type identifier of the first target database (e.g., the string "MySQL", "Oracle", or "PostgreSQL") as the only input parameter.

[0079] Step S2: Consistent Hash Mapping. The algorithm calculates the hash value of this type of identifier, obtaining a fixed numerical value. The platform maintains a virtual consistent hash ring with multiple virtual nodes evenly distributed on it. Each virtual node represents a logical position on the hash ring. The algorithm maps the calculated hash value to the virtual node on the ring and, starting from this position, searches clockwise along the ring to locate the first virtual node closest to the hash value.

[0080] Step S3: Locate the target behavior instance. The platform pre-configures a mapping relationship between virtual nodes and real SQL statement generation behavior instances (such as MySQLSQLGenerator, OracleSQLGenerator). A real behavior instance can be associated with multiple virtual nodes on the ring to achieve load balancing. Based on the virtual node located in Step S2, the algorithm queries the above mapping relationship to uniquely determine a target SQL statement generation behavior that needs to be invoked.

[0081] Step S4: Call and generate SQL. The algorithm does not directly handle the SQL generation logic, but instead calls the target SQL statement generation behavior located in step S3 from the instance resources maintained by the platform. After this behavior instance is called, its core logic begins: based on the received data management condition information (such as JSON statements) and its built-in syntax rules that completely match the first target database type, it generates a syntactically correct SQL statement that can be directly executed by the target database.

[0082] The significant advantage of this calling algorithm lies in its excellent scalability and system stability. When the data management platform needs to be expanded to support a new database type (e.g., introducing support for ClickHouse), the operations and maintenance personnel only need to perform the following operations: 1) Write the corresponding SQL statement generation behavior (such as ClickHouseSQLGenerator) for the new database type; 2) Select one or more free positions on the consistent hashing ring, add the corresponding new virtual nodes, and map them to the newly written behavior instances. Due to the characteristics of the consistent hashing algorithm, adding a new node only affects a small number of mapping relationships between the new node and the next existing node in the clockwise direction, while most of the original mapping relationships on the ring remain unchanged. This means that when the system routes requests for most existing database types, it will not be affected by the addition of new nodes, thus achieving smooth system expansion and high availability, and minimizing the disturbance to existing services during the expansion process.

[0083] In this way, each database type has a corresponding database interface, which can generate corresponding SQL statements based on the given database type. This reduces the coupling of internal code, simplifies the system's calling logic, and improves code reusability and scalability.

[0084] Furthermore, in the embodiments provided in this application, specifically, in order to construct and generate SQL statements for data management conditions, the query class corresponding to each database type can be dynamically set.

[0085] For example, continuing with the MySQL query class, for classes that inherit from the corresponding MySQL BaseQuery class, refer to... Figure 8 .

[0086] In this example, the SQL statement generation behavior can be regarded as a query behavior of the query class. The BaseQuery class uses the SQL generation interface to call the corresponding SQL statement generation behavior of MySQL. The corresponding query class of MySQL also inherits the MySQL syntax attributes of the BaseQuery class. In this way, the corresponding query class of MySQL can be constructed. This query class includes the SQL statement generation method and syntax attributes of MySQL. It generates the corresponding SQL statement according to the given database type, reduces the coupling of internal code, and improves the extensibility of the component.

[0087] As can be seen from the above, a unified SQL generation interface enables a unified calling method for different database operations, simplifies the system's calling logic, reduces the coupling between components in the data management platform, and improves code reusability and scalability.

[0088] Regarding the SQL statement generation behavior, the SQL statement generation behavior can be dynamically set. In this example, it is necessary to set the SQL statement generation behavior corresponding to each database type.

[0089] Reference Figure 9 In addition to generating SQL statements, query behaviors also include: obtaining database table names, obtaining column names of a table, and obtaining connectors. Each query behavior has a unified interface. Referring to the aforementioned method of constructing query classes, corresponding query classes can be constructed for multiple databases. By defining multiple query behaviors, the query classes can be extended, which is beneficial for further maintenance and progress of the system.

[0090] Taking retrieving database table names as an example, the data management platform defines a database table name interface, and the query class can be constructed in the following way: First, based on the database type, the corresponding behavior for retrieving the database table name is called, i.e., the behavior for retrieving the target database table name. The BaseQuery class corresponding to the database type executes the action of retrieving the database table name based on the target behavior. The query class constructed in the above way inherits the syntax attributes in the BaseQuery class and also includes the method for retrieving the database table name. Running this query class will retrieve the database table name.

[0091] Based on the aforementioned embodiments, in order to manage the database, it is necessary to set the data source information for each of the multiple databases.

[0092] Assume that the data management platform expects to manage databases A, B, C, and D. As mentioned above, A, B, C, and D correspond to multiple databases. Furthermore, it is necessary to set the data source information for A, B, C, and D.

[0093] Furthermore, a meta-mapping can be constructed using the database name and database access address. Specifically, a meta-mapping can be constructed using the database name and database port, or the database access address and database type. In this scenario, the data source information for each database is a meta-mapping constructed based on the database name, database access address, and database type.

[0094] The database constructed using the above method employs a meta-management and meta-mapping mechanism. The data management platform can automatically identify the database type of each data source and automatically identify and adapt the structure and characteristics of different data sources based on the data type.

[0095] Reference Figure 10 , Figure 10 Another method for managing multi-data source data provided in this application embodiment is applied to a data management platform. The data management platform is used to manage multiple databases and stores data source information for multiple databases. The method includes: S301: Select the second target database type and determine the second target database.

[0096] Here, the second target database type corresponds to the database selected by the user.

[0097] It should be noted that in some examples, the second target database type matches the first target database type. For example, if a user expects to query all database names under the MySQL data type, then MySQL corresponds to the second target database type, and all databases under the MySQL type will be displayed.

[0098] S302: Select the first target database.

[0099] S303: In response to a data management instruction, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data in the first target database.

[0100] Reference Figure 12 If the user selects MySQL as the second target database, the server will then display the data source name, database name, address, etc. of all databases.

[0101] The data source information of the first target database exists in the data source information of multiple databases, the second target database is at least a part of all databases that conform to the target database type, and the first target database exists in the second target database.

[0102] For S302 and S303, please refer to the steps of S101 and S102 mentioned above, which will not be repeated here.

[0103] It should be noted that the difference between the first target database type and the second target database type depends on the scenario, and the first target database type conforms to the second target database type.

[0104] For example, if there are three databases corresponding to MySQL, and the user needs to perform a query operation on database A, then, based on the above, database A corresponds to the first target database mentioned above. The database type of database A is MySQL, which is the first target database type MySQL.

[0105] The beneficial effects of the embodiments of this application are specifically referred to the foregoing and will not be repeated here.

[0106] As a preferred embodiment, in contrast to the foregoing, a connector can be used to determine a second target database. Specifically, here the connector is a query action, and the connector is specifically used to construct a mapping between a database type and the database corresponding to that database type.

[0107] In this embodiment, the data management platform has a defined connector interface. The aforementioned step S301: selecting the target database type and determining the second target database includes: S301a: Based on the second target database type, the connector interface calls the target connector.

[0108] It should be noted that the target connector is the connector corresponding to the second target database type.

[0109] S301b: Determine the second target database based on the target connector.

[0110] The target connector is built based on the second target database type and the database corresponding to the second target database type.

[0111] By using the above methods, the correct database connector can be dynamically selected based on the database type, providing the necessary flexibility and scalability for data query and filtering operations.

[0112] Furthermore, the data management platform is based on a unified data access standard, which defines a set of standard data operations.

[0113] The data management platform pre-configures several data operation executors, each of which implements the unified data access specification and is configured to provide specific functional logic for specific operations in the standard data operation.

[0114] When a data management instruction arrives at the platform, the data management platform responds by invoking the corresponding data operation executor that implements the unified data access specification. This invocation process specifically includes: First, the data management instructions are parsed to determine their operation type; Subsequently, based on the determined operation type, a specific data operation executor configured to handle that operation type is located from among several data operation executors that implement the unified data access specification; Finally, the specific data manipulation executor located is invoked.

[0115] During the execution of the data manipulation executor, it is configured to perform the following steps: Receive the data management condition information contained in the data management instruction; Based on the type of the first target database, adapt and call the statement generation logic corresponding to that type to generate operation statements that conform to the syntax of the first target database; The platform's data connector is invoked to establish a connection with the database using the data source information of the first target database, and the generated operation statements are executed, ultimately returning the operation results.

[0116] By introducing the unified data access standard and the corresponding data operation executor architecture, all data management instructions are routed to a unified processing flow, which decouples business logic from data access logic and enhances the system's cohesion, maintainability, and scalability.

[0117] Reference Figure 12 This application also provides a multi-data source management device for a data management platform, which stores data source information of multiple databases. Selected module: Used to select the first target database; Response module: Used to respond to data management instructions, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data in the first target database; The data source information for the first target database exists within the data source information of multiple databases.

[0118] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] Figure 13 A block diagram is shown that is suitable for implementing embodiments of the present invention. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0121] like Figure 13 As shown, the electronic device is represented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, memory 430, and communication bus 440 connecting different system components (including memory 430 and processing unit 410).

[0122] Communication bus 440 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0123] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0124] Memory 430 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0125] A program / utility having a set (at least one) of program modules can be stored in memory 430. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this invention.

[0126] Processor 410 executes various functional applications and data processing by running programs stored in memory 430, such as implementing embodiments of the present invention. Figures 1-11 The illustrated embodiment provides a method for managing data from multiple data sources.

[0127] The foregoing has described specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for managing data from multiple data sources, characterized in that, It is applied to a data management platform, which manages multiple databases. The data management platform stores data source information of multiple databases and is based on a unified data access specification. The unified data access specification defines a set of standard data operations, which are implemented by several data operation executors. Each data operation executor is configured to provide specific functional logic for a specific operation in the standard data operations. Select the first target database; In response to a data management command, a data operation executor that implements the unified data access specification and corresponds to the data management command is invoked; During the execution of the data operation executor, the first target database is accessed based on the data source information of the first target database, and corresponding operations are performed on the data in the first target database; The data source information of the first target database is contained within the data source information of the plurality of databases.

2. The method according to claim 1, characterized in that, Each of the multiple databases is a relational database. The data management platform is used to manage multiple databases of various relational databases. One database type corresponds to one SQL language format. The steps of responding to a data management instruction, accessing the first target database based on the data source information of the first target database, and performing corresponding operations on the data in the first target database include: In response to data management instructions, the data management condition information is converted into SQL statements for the data management conditions based on the syntax attributes corresponding to the first target database type. Access the first target database based on the data source information of the first target database, and perform corresponding operations on the first target database based on the SQL statement for the data management conditions; The data management condition information is contained within the data management instructions.

3. The method according to claim 2, characterized in that, The method further includes: Test whether the first target database is accessible based on the data source information of the first target database; If accessible, the steps of accessing the first target database based on the data source information of the first target database and performing corresponding operations on the first target database based on the SQL statement for data management conditions are executed. If the access is denied, an access denied message will be generated.

4. The method according to claim 2, characterized in that, The data management instructions include: data sorting instructions, and the data management information includes: filtering and sorting information; and / or The data management instructions include: data aggregation instructions, and the data management condition information includes: data aggregation condition information; and / or The data management instructions include: data query instructions, and the data management condition information includes: data query condition information; and / or The data management instructions include: a data deduplication instruction, and the data management condition information corresponds to the data deduplication instruction.

5. The method according to claim 2, characterized in that, The data management condition information is in the form of a JSON statement. The SQL statement for the data management conditions is: an SQL statement generated based on the syntax attributes corresponding to the first target database type and the JSON statement corresponding to the data management condition information; The data management platform has a defined SQL generation interface, which sets the SQL statement generation behavior for each database type. The SQL statement for the data management conditions is generated as follows: Based on the first target database type, the SQL generation interface calls the target to generate SQL statements. Based on the target, the SQL statement is generated, and based on the data management condition information and the syntax attributes corresponding to the first target database type, the SQL statement for the data management conditions is generated.

6. The method according to claim 5, characterized in that, The SQL generation interface calls the target to generate SQL statements, which is achieved by calling an algorithm, the calling algorithm including the following steps: S1: Use the type identifier of the first target database as the input parameter; S2: Calculate the hash value of the type identifier, and based on the consistent hashing algorithm, map the hash value to a virtual node ring to determine the nearest virtual node in the clockwise direction along the ring; S3: Determine the corresponding target SQL statement generation behavior based on the preset mapping relationship between virtual nodes and SQL statement generation behavior; S4: Invoke the determined target SQL statement generation behavior to generate an SQL statement that conforms to the syntax of the first target database; When the data management platform is expanded to support new database types, the impact on the mapping relationship between existing database types and SQL statement generation behavior is minimized by adding corresponding virtual nodes and SQL statement generation behavior for the new database types.

7. The method according to claim 1, characterized in that, The invocation of the data operation executor that corresponds to the data management instruction and implements the unified data access standard includes: The data management instructions are parsed to determine their operation type; Based on the operation type, from among several data operation executors that implement the unified data access specification, a specific data operation executor configured to handle that operation type is located; Invoke the specific data operation executor located.

8. The method according to claim 7, characterized in that, The execution process of the invoked specific data manipulation executor includes: The data operation executor receives data management condition information from the data management instruction; The data operation executor adapts to and calls the statement generation logic corresponding to the type of the first target database to generate operation statements that conform to the syntax of the first target database. The data operation executor calls the data connector, establishes a connection using the data source information of the first target database, and executes the generated operation statements.

9. The method according to claim 1, characterized in that, The method further includes: setting data source information for each of the plurality of databases; The data source information for each database is a meta-mapping constructed based on the database name, database access address, and database type.

10. A management device for multi-source data, characterized in that, It is applied to a data management platform, which stores data source information from multiple databases; Selected module: Used to select the first target database; Response module: Used to respond to data management instructions, access the first target database based on the data source information of the first target database, and perform corresponding operations on the data in the first target database; The data source information of the first target database is contained within the data source information of the plurality of databases.

Images