Data mobilization method and apparatus, storage medium, and electronic device

Abstract

The application discloses a data mobilization method and device, a storage medium and an electronic device, relates to the technical field of big data, and the data mobilization method comprises the following steps: in the case that the query information sent by a client is acquired, field information corresponding to the query information is recognized, the database type of a target database responding to the query information is determined according to the field information, and a metadata set corresponding to the database type is acquired; a table building mode is started, data information in the metadata set is mapped into a first database table in a preset format corresponding to the target database to obtain a first table, and the data information in the metadata set is mapped into a second database table in a data warehouse system to obtain a second table; in the case that an association relationship of the same data information between the first table and the second table is established, the second table is sent to the client to indicate data mobilization responding to the query information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a data mobilization method and device, a storage medium and an electronic device. BACKGROUND

[0002] For the problem of reading an MPP (massively parallel processing) database by a big data computing engine, in the related art, as shown in the following formula (1), mainly through the computing engine in the field of big data such as Spark, Presto, etc., the MPP database is connected through the jdbc (Java DataBase Connectivity) protocol, and then the data is read, but the above-mentioned method has poor performance in reading the MPP database, and the performance in the data reading stage is limited by the performance bottleneck of the single-threaded data output of the coordination node, especially when the data volume is very large, the reading time-consuming will be very long. In addition, the above-mentioned method cannot effectively play the concurrent processing characteristics of the computing engine such as Spark, Presto, etc., and is limited by the mechanism of single-threaded data pulling of jdbc, so that the data computing task can only be allocated to a certain working node, resulting in idle resources of other computing nodes, and the ability of distributed computing cannot be fully utilized. Figure 1

[0003] In view of the problems in the related art that the multi-concurrent processing of the database in the whole data reading link cannot be realized, and the data processing efficiency is low, etc., no effective solution has been proposed so far.

[0004] Therefore, it is necessary to improve the related art to overcome the defects in the related art. SUMMARY

[0005] The embodiments of the present application provide a data mobilization method and device, a storage medium and an electronic device to at least solve the problems that the multi-concurrent processing of the database in the whole data reading link cannot be realized, and the data processing efficiency is low, etc.

[0006] ​According to an aspect of the embodiments of the present application, a data mobilization method is provided, comprising: in the case that query information sent by a client is acquired, identifying field information corresponding to the query information, wherein the field information is used to indicate a target file matched in a target database; determining a database type of the target database responding to the query information according to the field information, and acquiring a metadata set corresponding to the database type; starting a table building mode, mapping data information in the metadata set to a first database table in a preset format corresponding to the target database to obtain a first table, and mapping data information in the metadata set to a second database table in a data warehouse system to obtain a second table; in the case that an association relationship for the same data information is established between the first table and the second table, sending the second table to the client to indicate data mobilization responding to the query information.

[0007] In an exemplary embodiment, before determining the database type of the target database responding to the query information according to the field information, the above method further comprises: parsing the field information to obtain a library name and a table name corresponding to the target database; determining the target database from a database set according to the library name and the table name; and parallel exporting data stored in a data node of the target database to a cache space corresponding to the client.

[0008] In an exemplary embodiment, before parallel exporting data stored in the data node of the target database to the cache space corresponding to the client, the above method further comprises: determining a mounting record of a cache component corresponding to the client; in the case that the mounting record indicates that the client has the cache component, determining directory information corresponding to a cache space set in the client by the cache component, and generating a data file address corresponding to the directory information; in the case that the mounting record indicates that the client does not have the cache component, determining that data exporting of the target database is abnormal, and generating corresponding operation information.

[0009] In an exemplary embodiment, after determining the directory information corresponding to the cache space set in the client by the cache component, and generating the data file address corresponding to the directory information, the above method further comprises: determining a storage quantity of the data file address for exporting data of the target database; in the case that the storage quantity is greater than a preset quantity, instructing the cache component to set a new cache space in the client; in the case that the storage quantity is less than or equal to the preset quantity, determining whether the target database completes the operation of exporting data.

[0010] In an example embodiment, determining whether the target database completes the operation of exporting data comprises: in a case where it is determined that the target database completes the operation of exporting data, allowing the client to respond to the query information according to data in the cache space; in a case where it is determined that the target database does not complete the operation of exporting data, prohibiting the client from responding to the query information according to data in the cache space.

[0011] In an example embodiment, after the second table is sent to the client to indicate the data mobilization in response to the query information, the method further comprises: determining an association relationship between the second table and a data file address existing in the client; and generating a distributed query rule corresponding to the target database according to the association relationship.

[0012] In an example embodiment, after the second table is sent to the client to indicate the data mobilization in response to the query information, the method further comprises: obtaining a response result of the second table to the query information; in a case where the response result indicates that the data mobilization in response to the query information has been successfully indicated, obtaining a time at which the second table responds to the query information again, determining a time difference value of the second table responding to the query information twice, and in a case where the time difference value is greater than a preset difference value, sending an operation instruction of deleting the second table to the client; in a case where the response result indicates that the data mobilization in response to the query information has not been successfully indicated, updating a metadata set of the second table.

[0013] According to another aspect of the embodiments of the present application, a data mobilization apparatus is further provided, comprising: an obtaining module, configured to, in a case where query information sent by a client is obtained, identify field information corresponding to the query information, wherein the field information is used to indicate a target file matched in a target database; a determining module, configured to determine a database type of a target database responding to the query information according to the field information, and obtain a metadata set corresponding to the database type; a mapping module, configured to start a table building mode, map data information in the metadata set into a first database table in a preset format corresponding to the target database to obtain a first table, and map the data information in the metadata set into a second database table in a data warehouse system to obtain a second table; and a sending module, configured to, in a case where an association relationship between the first table and the second table for the same data information is established, send the second table to the client to indicate the data mobilization in response to the query information.

[0014] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is configured to execute the data mobilization method when running.

[0015] According to another aspect of the embodiments of the present application, an electronic device is provided, and the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the data mobilization method through the computer program.

[0016] According to the present application, when the query information sent by the client is obtained, the field information corresponding to the query information is identified, wherein the field information is used to indicate the target file matched in the target database; the database type of the target database responding to the query information is determined according to the field information, and the metadata set corresponding to the database type is obtained; the table building mode is started, the data information in the metadata set is mapped into the first database table in the preset format corresponding to the target database to obtain the first table, and the data information in the metadata set is mapped into the second database table in the data warehouse system to obtain the second table; when the association relationship of the same data information between the first table and the second table is established, the second table is sent to the client to indicate the data mobilization responding to the query information, so as to solve the problems that the multi-concurrent processing of the database in the whole data reading link cannot be realized, the data processing efficiency is low, and the like. Through the query conversion between the first table and the second table, the specific query process of the query information is converted into the distributed query based on the second table (Hive external table), the query efficiency is improved, and the multi-concurrent processing of the database in the whole data reading link is realized. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles behind the application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0019] Figure 1 is an architecture schematic diagram of data mobilization in a MPP database according to the related art;

[0020] Figure 2 is a hardware environment schematic diagram of the data mobilization method according to the embodiments of the present application;

[0021] Figure 3 is a flow chart of a data mobilization method according to an embodiment of the present application;

[0022] Figure 4 is a whole architecture diagram of a data mobilization according to an optional embodiment;

[0023] Figure 5 is a whole architecture diagram of another data mobilization according to an optional embodiment;

[0024] Figure 6 is a flow chart of a data mobilization method according to an embodiment of the present application;

[0025] Figure 7 is a structure block diagram of a data mobilization apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] The present application will be described below in conjunction with the embodiments:

[0029] According to an aspect of an embodiment of the present application, a data mobilization method is provided. Optionally, in the present embodiment, the above-mentioned data mobilization method can be applied to a hardware environment composed of a server 101 and a big data computing engine 103 as shown in Figure 2 . As shown in Figure 2As shown, server 101 is connected to big data computing engine 103 via a network and can be used to provide services for big data computing engine 103 or application 107 installed on big data computing engine 103. Application 107 can be a data mobilization application, etc. Database 105 can be set up on server 101 or independently of server 101 to provide data storage services for server 101, such as vehicle data storage server or environmental data storage server. The network can include, but is not limited to, wired network and wireless network. The wired network includes local area network, metropolitan area network and wide area network. The wireless network includes Bluetooth, WIFI and other networks that enable wireless communication. Big data computing engine 103 can be a terminal configured with an application, which can include, but is not limited to, a mobile terminal. The server 101 can be a single server, a server cluster composed of multiple servers, or a cloud server. Application 107 using the above data mobilization method is displayed through big data computing engine 103 or other connected display devices.

[0030] Combination Figure 2 As shown, the above data mobilization method can be implemented in the big data computing engine 103 through the following... Figure 2 Steps S202-S206 are implemented as follows:

[0031] Optionally, in this embodiment, the above-described data retrieval method can also be implemented via a server, for example, Figure 2 It is implemented in server 101 shown; or it is implemented jointly by the big data computing engine and the server.

[0032] The above is merely an example, and this embodiment does not impose any specific limitations.

[0033] Alternatively, as an alternative implementation method, Figure 3 This is a flowchart of a data retrieval method according to an embodiment of this application, such as... Figure 3 As shown, the above data retrieval method includes the following steps S202-S208:

[0034] Step S202: Upon receiving the query information sent by the client, identify the field information corresponding to the query information, wherein the field information is used to indicate the matching of the target file in the target database;

[0035] Step S204: Determine the database type of the target database responding to the query information based on the field information, and obtain the metadata set corresponding to the database type;

[0036] Optionally, the above metadata set includes at least one of the following: the table name corresponding to the database type, the fields of the table corresponding to the database type, and the field types corresponding to the fields of the table.

[0037] Step S206, start the table building mode, map the data information in the metadata set to the first database table corresponding to the preset format of the target database to obtain a first table, and map the data information in the metadata set to a second database table in the data warehouse system to obtain a second table;

[0038] As an optional example, in actual application, the first table can be a database table corresponding to a GaussDB database, and the second table can be a Hive external table, which is used to indicate that the data in the GaussDB database is exported in parallel to the external cache corresponding to the database after the query is initiated, and the external addresses corresponding to different data.

[0039] Step S208, in the case of establishing the association relationship between the first table and the second table for the same data information, the second table is sent to the client to indicate the data mobilization responding to the query information.

[0040] It should be noted that the client can be a mobile phone, a tablet, a computer, or other terminal devices.

[0041] Through the present application, in the case of obtaining the query information sent by the client, the field information corresponding to the query information is identified, wherein the field information is used to indicate matching a target file in a target database; the database type of the target database responding to the query information is determined according to the field information, and the metadata set corresponding to the database type is obtained; a table building mode is started, the data information in the metadata set is mapped to the first database table corresponding to the preset format of the target database to obtain a first table, and the data information in the metadata set is mapped to a second database table in the data warehouse system to obtain a second table; in the case of establishing the association relationship between the first table and the second table for the same data information, the second table is sent to the client to indicate the data mobilization responding to the query information, thereby solving the problems of being unable to implement multi-concurrent processing of the database in the entire data reading link, and low data processing efficiency, etc., and through the query conversion between the first table and the second table, the specific query process of the query information is converted into a distributed query based on the second table (Hive external table), the query efficiency is improved, and multi-concurrent processing of the database in the entire data reading link is realized.

[0042] In an example embodiment, before determining the database type of the target database responding to the query information according to the field information, the method further comprises: parsing the field information to obtain the library name and the table name corresponding to the target database; determining the target database from the database set according to the library name and the table name; and storing the data of the data nodes of the target database in parallel to the cache space corresponding to the client.

[0043] It can be understood that the library name of the target database storing the corresponding data and the table name of the data table corresponding to the target database can also be determined according to the field information carried in the query information. Based on the library name and the table name, the target database can be quickly located, so that the data stored by different nodes in the target database is exported in parallel to the distributed file system or the distributed cache outside the target database in combination with the library name and the table name. The distributed characteristics are utilized to enable these exported data to perform multi-concurrent processing in the entire data reading link, greatly improving the data processing efficiency. In addition, if the data is exported to the external distributed cache, the client does not need to interact with the database when responding to the query information, and the query information of the client can be responded to only according to the data in the distributed cache. Therefore, the response time of the query information is improved, and the use experience of the target user is improved.

[0044] It should be noted that the high-speed protocol or middleware of the database (such as the gds middleware of GaussDB, the gpfdist of Greenplum, the fdw mechanism of Teradata, etc.) can be used for parallel export. Specifically, the fast export protocol will use the middleware to concurrently export the data from the data nodes of the database to the external file system. The external file system can be a distributed file system or a distributed cache in contact with the client.

[0045] In an example embodiment, before storing the data of the data nodes of the target database in parallel to the cache space corresponding to the client, the method further comprises: determining the mounting record of the cache component corresponding to the client; in the case that the mounting record indicates that the client has a cache component, determining the directory information corresponding to the cache space of the client in the cache component, and generating a data file address corresponding to the directory information; in the case that the mounting record indicates that the client does not have a cache component, determining that the data export of the target database is abnormal, and generating corresponding operation information.

[0046] It can be understood that, in order to improve the read-write efficiency of the database export data, the read-write efficiency of the export data can also be improved by mounting the corresponding cache component in the cache space corresponding to the distributed file system or the distributed cache. For example, the read-write efficiency can be improved by means of the distributed cache system Alluxio. The Alluxio component is mounted to the local directory, and when the data is exported to the local directory, the data is written to Alluxio through the Alluxio fuse mounting mechanism. Then, when creating the external table of Hive, the location of the external table is specified to the specific path of the data file in Alluxio. Finally, the current query is converted into a query on the Hive table. When the big data engine reads the Hive table, it obtains the data file address of the table through the Hive metadata, so that the big data engine can fully mobilize the various worker nodes to read the data on the various worker nodes of Alluxio in parallel, thereby truly realizing the concurrent output of the MPP database. The big data computing engine reads the data in parallel, and truly realizes distributed data processing.

[0047] In an exemplary embodiment, after determining the directory information corresponding to the cache space set by the cache component in the client and generating the data file address corresponding to the directory information, the method further comprises: determining the storage quantity of the data file address for the target database export data; in the case that the storage quantity is greater than a preset quantity, instructing the cache component to set a new cache space in the client; in the case that the storage quantity is less than or equal to the preset quantity, determining whether the target database completes the export data operation.

[0048] That is, since the data file address may have a certain data storage quantity, when the data cached from the target database exceeds the storage quantity of the current data file address, a new storage space can be set to ensure the smooth progress of data parallel export.

[0049] In an exemplary embodiment, determining whether the target database completes the export data operation comprises: in the case that it is determined that the target database completes the export data operation, allowing the client to respond to the query information according to the data in the cache space; in the case that it is determined that the target database does not complete the export data operation, prohibiting the client from responding to the query information according to the data in the cache space.

[0050] In order to ensure effective response to the query information and avoid poor query results caused by incomplete data, it is necessary to ensure that all data in the target database are imported before responding to the query information through the data in the cache space.

[0051] In an exemplary embodiment, after the second table is sent to the client to indicate the data mobilization in response to the query information, the method further comprises: obtaining a response result of the second table to the query information; in a case where the response result indicates that the data mobilization in response to the query information has been successfully indicated, obtaining a time at which the second table responds to the query information again, and determining a time difference value of the second table responding to the query information twice, and in a case where the time difference value is greater than a preset difference value, sending an operation instruction of deleting the second table to the client; in a case where the response result indicates that the data mobilization in response to the query information has not been successfully indicated, updating a metadata set of the second table.

[0052] In an exemplary embodiment, after the second table is sent to the client to indicate the data mobilization in response to the query information, the method further comprises: obtaining a response result of the second table to the query information; in a case where the response result indicates that the data mobilization in response to the query information has been successfully indicated, obtaining a time at which the second table responds to the query information again, and determining a time difference value of the second table responding to the query information twice, and in a case where the time difference value is greater than a preset difference value, sending an operation instruction of deleting the second table to the client; in a case where the response result indicates that the data mobilization in response to the query information has not been successfully indicated, updating a metadata set of the second table.

[0053] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. In order to better understand the above method, the above process is described in combination with the embodiments as follows, but is not used to limit the technical solutions of the embodiments of the present application, specifically:

[0054] In an optional embodiment, Figure 4 is a whole architecture diagram of data mobilization according to an optional embodiment, as Figure 4 shown: a multi-connection mode is created to improve the concurrency of the engine reading the MPP database. The specific technical solutions are as follows: the table to be queried is divided according to a certain key, and it is emphasized that a numerical field is selected as the partition key here, so that the range division can be performed according to the expected concurrent data. After the table is divided into multiple shards, a jdbc connection is established for each shard, and each connection reads the data of a specific range of shards, thereby realizing the concurrent reading of the target table by the engine. However, the above scheme has the following problems when applied: first, the field type requirement is high: the table needs to have a numerical field, and if not, effective division cannot be performed; second, the field value requirement is high: for numerical fields, the values need to be continuous, and if the field value distribution is sparse, the data volume of the divided data shards will be uneven, and the data skew problem may occur; third, pressure is caused to the data source system: if the concurrency is configured too high, a problem of creating dozens or even hundreds of jdbc connections to query a table will occur, which will cause a certain access pressure to the MPP data source. In addition, the above technical solution has not solved the problem of concurrent "data output" on the data source side, so that the single-concurrent bottleneck still exists in the whole data reading link.

[0055] In order to avoid the above problems, as an optional embodiment, the application also provides another overall architecture diagram of data mobilization, Figure 5 is another overall architecture diagram of data mobilization according to an optional embodiment. Specifically, the query of the MPP database is converted into a query of the Hive external table, that is, the table data of the MPP database is quickly exported to an external distributed file system or distributed cache through a fast export protocol or middleware of the MPP database, then the data file path of the exported data is bound by creating a Hive external table, and finally the Hive table data is read through a big data computing engine.

[0056] It should be noted that when the user queries the MPP database, the fast export protocol of the MPP database will utilize the middleware to concurrently export data from the data nodes of the MPP database to the external file system in parallel. In order to reduce the efficiency of data export, a distributed cache system Alluxio can also be used to improve the read-write efficiency. The Alluxio component is mounted to the local directory, and when the data is exported to the local directory, the data is written to Alluxio through the Alluxio fuse mounting mechanism. Then when creating the Hive external table, the location of the external table is specified to the specific path of the data file in Alluxio. Finally, the query is converted into a query of the Hive table. When the big data engine reads the Hive table, the data file address of the table is obtained through the Hive metadata, so that the working nodes of the big data engine can be fully mobilized to concurrently read the data on the working nodes of Alluxio, thereby truly realizing concurrent output of data of the MPP database, parallel reading of data by multiple nodes of the big data computing engine, and truly distributed data processing.

[0057] Optionally, Figure 6 is a flowchart of data mobilization according to an optional embodiment. Taking GaussDB as an example, the database includes the following steps:

[0058] Before executing the data query service, start the fuse application program corresponding to the external data storage system, and mount the local GDS target directory through the fuse application program.

[0059] Step 1, initiate a SQL query;

[0060] Step 2, obtain the library (GS, Hive) name and table name;

[0061] Step 3, obtain the schema information according to the library name and table name;

[0062] Step 4, return the table in the metadata information;

[0063] Step 5, enter the table building mode, and map the obtained metadata information into GaussDB table and Hive table respectively;

[0064] Step 6, create a Hive external table mounted to the table directory (jdbc) of the Alluxio fuse connection

[0065] Step 7, asynchronously trigger the GSGDS operation;

[0066] Step 7.1, create a GS external table mounted to the Alluxio fuse mounted directory;

[0067] Step 7.2, execute the insert operation to trigger the GDS derivative action, and then import the data in the database into the table directory corresponding to the Alluxio fuse mounted directory in multiple parallel ways.

[0068] Step 7.3, data import, end, and state modification;

[0069] Step 8, state polling;

[0070] Step 9, query the GS2 Hive table through the Hive Connector;

[0071] Step 10, return the query result;

[0072] Step 11, return the result;

[0073] Optionally, in the application, the above steps are implemented in the following manner: when a user initiates a query statement for MPP database query, the schema information of the GaussDB database table corresponding to the GaussDB is first queried, and then the fields of the GaussDB table and the Hive table are mapped according to the information, and the statement of the external table is spliced according to the template. When the query enters the metadata checking stage, if the table to be queried does not exist in the Hive, the Hive external table is created at this time, and the path points to the table directory of Alluxio (the directory is agreed in advance, / gdsdata / tblname). Next, the GaussDB external table is created, and the path of the external table is specified to a certain external address (the address is mounted in advance by Alluxio fuse) by means of gds server (a lightweight service implemented based on the fdw mechanism of GaussDB). The data is exported to the external address located by the GaussDB external table through the insertinto statement. Since the external address has been mounted by Alluxio, the data actually enters Alluxio. After the data export is completed, the entire process is notified to continue, thereby completing the entire query scheduling. It should be noted that the Hive Connector is the general term for the process of reading the Hive table by the big data engine.

[0074] It should be noted that the above method converts the jdbc query of the traditional MPP database into a distributed query of the Hive external table, improves the query efficiency, and exports the data of the MPP database to the distributed cache system to reduce the disk IO overhead. Optionally, the Alluxio is a distributed file system based on memory, and if the performance requirement is not strict, the HDFS distributed file system can be used instead.

[0075] Through the above steps, the data source is concurrently exported from each data node to the distributed file system, and then the distributed big data computing engine reads and processes the data in parallel, which realizes distributed data processing on the big data computing engine side and concurrent data output on the MPP data source side, and can improve the query performance of the engine side and the MPP database side at the same time. In addition, the above implementation does not involve table splitting, so it does not involve restrictions on the fields and data of the table. Further, the data nodes of the MPP database are read by the big data computing engine, which avoids the pressure on the coordination node of the MPP database, and greatly improves the processing efficiency of the entire scheduling mechanism and scheduling process.

[0076] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a general hardware platform as necessary, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part of the prior art that makes a contribution, and the computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method of each embodiment of the present application.

[0077] In the present embodiment, a data mobilization apparatus is also provided, which is used to implement the above embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware, or a combination of software and hardware is also possible and is contemplated.

[0078] Figure 7 is a structural block diagram of a data mobilization apparatus according to an embodiment of the present application, which includes:

[0079] The acquisition module 72 is configured to, in a case where the query information sent by the client is acquired, identify field information corresponding to the query information, wherein the field information is used to indicate a target file matched in a target database;

[0080] The determination module 74 is configured to determine a database type of the target database responding to the query information according to the field information, and acquire a metadata set corresponding to the database type;

[0081] The mapping module 76 is configured to start a table building mode, map data information in the metadata set into a first database table in a preset format corresponding to the target database to obtain a first table, and map the data information in the metadata set into a second database table in a data warehouse system to obtain a second table;

[0082] The sending module 78 is configured to, in a case where an association relationship for the same data information between the first table and the second table is established, send the second table to the client to indicate data mobilization responding to the query information.

[0083] Through the above device, in the case that the query information sent by the client is acquired, the field information corresponding to the query information is identified, wherein the field information is used to indicate the target file matched in the target database; the database type of the target database responding to the query information is determined according to the field information, and the metadata set corresponding to the database type is acquired; the table building mode is started, the data information in the metadata set is mapped into the first database table in the preset format corresponding to the target database to obtain the first table, and the data information in the metadata set is mapped into the second database table in the data warehouse system to obtain the second table; in the case that the association relationship of the same data information between the first table and the second table is established, the second table is sent to the client to indicate the data mobilization responding to the query information, so as to solve the problems that the multi-concurrent processing of the database in the whole data reading link cannot be realized, the data processing efficiency is low and the like, the specific query process of the query information is converted into the distributed query based on the second table (Hive external table) through the query conversion between the first table and the second table, the query efficiency is improved, and the multi-concurrent processing of the database in the whole data reading link is realized.

[0084] In an exemplary embodiment, the above device further comprises: a derivation module, configured to, before determining the database type of the target database responding to the query information according to the field information, parse the field information to obtain the library name and the table name corresponding to the target database; determine the target database from the database set according to the library name and the table name; and parallelly derive the data stored in the data node of the target database to the cache space corresponding to the client.

[0085] In an exemplary embodiment, the above derivation module further comprises: a component unit, configured to, before parallelly deriving the data stored in the data node of the target database to the cache space corresponding to the client, determine the mounting record of the cache component corresponding to the client; in the case that the mounting record indicates that the client has the cache component, determine the directory information corresponding to the cache space of the cache component set in the client and generate the data file address corresponding to the directory information; in the case that the mounting record indicates that the client does not have the cache component, determine that the data derivation of the target database is abnormal and generate the corresponding operation information.

[0086] In an example embodiment, the deriving module further includes a storage unit configured to determine directory information corresponding to the cache space set by the cache component in the client, and generate a data file address corresponding to the directory information, and determine a storage amount of the data file address for the target database to derive data; in a case where the storage amount is greater than a preset amount, instruct the cache component to set a new cache space in the client; in a case where the storage amount is less than or equal to the preset amount, determine whether the target database completes the operation of deriving data.

[0087] In an example embodiment, the storage unit is further configured to, in a case where it is determined that the target database completes the operation of deriving data, allow the client to respond to the query information according to data in the cache space; in a case where it is determined that the target database does not complete the operation of deriving data, prohibit the client from responding to the query information according to data in the cache space.

[0088] In an example embodiment, the apparatus further includes a rule module configured to, in a case where the association relationship between the first table and the second table for the same data information is established, after the second table is sent to the client, determine an association relationship between the second table and a data file address existing in the client; and generate a distributed query rule corresponding to the target database according to the association relationship.

[0089] In an example embodiment, the apparatus further includes an updating module configured to obtain a response result of the second table to the query information; in a case where the response result indicates that the data movement of the query information has been successfully instructed, obtain a time at which the second table responds to the query information again, and determine a time difference value of the second table responding to the query information twice, in a case where the time difference value is greater than a preset difference value, send an operation instruction of deleting the second table to the client; in a case where the response result indicates that the data movement of the query information has not been successfully instructed, update a metadata set for generating the second table.

[0090] Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the steps in any of the method embodiments described above when running.

[0091] Optionally, in the present embodiment, the storage medium can be configured to store a computer program for executing the following steps:

[0092] S1, in the case of obtaining the query information sent by the client, identifying field information corresponding to the query information, wherein the field information is used to indicate matching a target file in a target database;

[0093] S2, determining a database type of a target database responding to the query information according to the field information, and obtaining a metadata set corresponding to the database type;

[0094] S3, starting a table building mode, mapping data information in the metadata set to a first database table in a preset format corresponding to the target database to obtain a first table, and mapping the data information in the metadata set to a second database table in a data warehouse system to obtain a second table;

[0095] S4, in the case of establishing an association relationship between the first table and the second table for the same data information, sending the second table to the client to indicate data mobilization responding to the query information.

[0096] In one example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various computer program storage media.

[0097] The specific examples in the embodiment can refer to the examples described in the above embodiments and example embodiments, which will not be described here again.

[0098] The embodiment of the present application also provides an electronic device, including a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0099] Optionally, in the embodiment, the above processor can be configured to execute the following steps through the computer program:

[0100] S1, in the case of obtaining the query information sent by the client, identifying field information corresponding to the query information, wherein the field information is used to indicate matching a target file in a target database;

[0101] S2, determining a database type of a target database responding to the query information according to the field information, and obtaining a metadata set corresponding to the database type;

[0102] S3, starting a table building mode, mapping data information in the metadata set into a first database table in a preset format corresponding to the target database to obtain a first table, and mapping data information in the metadata set into a second database table in the data warehouse system to obtain a second table;

[0103] S4, in the case of establishing an association relationship between the first table and the second table for the same data information, sending the second table to the client to instruct data mobilization in response to the query information.

[0104] In one example embodiment, the electronic device described above can further include a transmission device connected to the processor and an input / output device connected to the processor.

[0105] The specific examples in the embodiments can refer to the examples described in the above embodiments and exemplary embodiments, which will not be described here.

[0106] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, which can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0107] The above is only the preferred embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A data mobilization method characterized by, The application is applied in a big data computing engine, comprising: In the case of obtaining the query information sent by the client, the field information corresponding to the query information is identified, wherein the field information is used to indicate the matching target file in the target database; According to the field information, the database type of the target database responding to the query information is determined, and the metadata set corresponding to the database type is obtained; Start the table building mode, map the data information in the metadata set to the first database table in the target database corresponding to the preset format to obtain the first table, and map the data information in the metadata set to the second database table in the data warehouse system to obtain the second table; In the case of establishing the association relationship between the first table and the second table for the same data information, the second table is sent to the client to indicate the data mobilization responding to the query information; Before determining the database type of the target database responding to the query information according to the field information, the method further comprises: parsing the field information to obtain the library name and table name corresponding to the target database; determining the target database from the database set according to the library name and table name; and parallel exporting the data stored in the data node of the target database to the distributed cache space corresponding to the client; Before parallel exporting the data stored in the data node of the target database to the distributed cache space corresponding to the client, the method further comprises: determining the mounting record of the cache component corresponding to the client; in the case that the mounting record indicates that the client has a cache component, determining the directory information corresponding to the distributed cache space set in the client by the cache component, and generating a data file address corresponding to the directory information; in the case that the mounting record indicates that the client does not have a cache component, determining that the data export of the target database is abnormal, and generating corresponding operation information; After sending the second table to the client in the case of establishing the association relationship between the first table and the second table for the same data information, the method further comprises: determining the association relationship between the second table and the data file address existing in the client; and generating the distributed query rule corresponding to the target database according to the association relationship.

2. The data mobilization method of claim 1, wherein, After determining the directory information corresponding to the distributed cache space set in the client by the cache component, and generating a data file address corresponding to the directory information, the method further comprises: Determining the storage quantity of the data file address for the exported data of the target database; In the case that the storage quantity is greater than the preset quantity, instructing the cache component to set a new distributed cache space in the client; In the case that the storage quantity is less than or equal to the preset quantity, determining whether the target database completes the data export operation.

3. The data mobilization method of claim 2, wherein, Determining whether the target database completes the data export operation comprises: in a case where it is determined that the target database completes the operation of exporting data, allowing the client to respond to the query information according to data in the distributed cache space; in a case where it is determined that the target database does not complete the operation of exporting data, prohibiting the client from responding to the query information according to data in the distributed cache space.

4. The data mobilization method of claim 1, wherein, After sending the second table to the client to indicate the data mobilization responding to the query information, the method further comprises: obtaining a response result of the second table to the query information; in a case where the response result indicates that the data mobilization responding to the query information has been successfully indicated, obtaining a time at which the second table responds to the query information again, determining a time difference value of the second table responding to the query information twice, and in a case where the time difference value is greater than a preset difference value, sending an operation instruction of deleting the second table to the client; in a case where the response result indicates that the data mobilization responding to the query information has not been successfully indicated, updating a metadata set of generating the second table.

5. A data mobilization apparatus, characterized by, The application is applied in a big data computing engine, comprising: an obtaining module, configured to, in a case where query information sent by a client is obtained, identify field information corresponding to the query information, wherein the field information is used to indicate a target file matched in a target database; a determining module, configured to determine a database type of a target database responding to the query information according to the field information, and obtain a metadata set corresponding to the database type; a mapping module, configured to start a table building mode, map data information in the metadata set to a first database table in a preset format corresponding to the target database to obtain a first table, and map the data information in the metadata set to a second database table in a data warehouse system to obtain a second table; a sending module, configured to, in a case where an association relationship for the same data information between the first table and the second table is established, send the second table to the client to indicate the data mobilization responding to the query information; the device further comprises an exporting module, configured to, before determining the database type of the target database responding to the query information according to the field information, parse the field information to obtain a library name and a table name corresponding to the target database, determine the target database from a database set according to the library name and the table name, and parallelly export data stored in a data node of the target database to a distributed cache space corresponding to the client. The export module further comprises a component unit, configured to determine mounting records of cache components corresponding to the client before parallel exporting data stored in data nodes of the target database to the distributed cache space corresponding to the client; in the case that the mounting records indicate that the client has cache components, determine directory information corresponding to the distributed cache space set in the client by the cache components, and generate a data file address corresponding to the directory information; in the case that the mounting records indicate that the client does not have cache components, determine that data export of the target database is abnormal, and generate corresponding operation information. The apparatus further comprises a rule module, configured to, in the case that an association relationship for the same data information between the first table and the second table is established, determine an association relationship between the second table and a data file address existing in the client after sending the second table to the client; and generate a distributed query rule corresponding to the target database according to the association relationship.

6. A computer readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program performs the method of any one of claims 1 to 4 when executed. 7.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 4 by using the computer program.

Images