Data processing method, electronic device and storage medium

Abstract

The present invention discloses a data processing method, an electronic device and a storage medium, wherein the data processing method comprises: obtaining data source information and first metadata information of multiple data sources, and storing the data source information and the first metadata information of each data source in a database; in response to the update of the data source information of at least one data source, obtaining second metadata information corresponding to the updated data source information; using the second metadata information to replace the first metadata information corresponding to the same data source in the database; that is, in the present application, after the data source information of the data source is updated, the metadata information corresponding to the updated data source information can be obtained, and then the metadata information of the same data source in the database can be updated, which can effectively ensure the consistency of the metadata information stored in the database with the metadata information corresponding to the corresponding data source.

Description

Technical Field

[0001] The present application relates to the technical field of big data analysis of data sources, and in particular to a data processing method, electronic device and storage medium. Background Art

[0002] With the explosive growth of computer applications, the requirements for data management are becoming increasingly higher. For different data sources in various industries, there is corresponding metadata information, and database query systems focusing on different fields have emerged. Each database query system has independent metadata information management services.

[0003] In practice, as data sources are constantly changing and updated, the corresponding metadata information also changes. However, the current database query system cannot guarantee the consistency between the stored metadata information and the metadata information corresponding to the corresponding data source, which affects data management and analysis, and in turn affects data query efficiency. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a data processing method, an electronic device and a storage medium, which can obtain metadata information corresponding to the updated data source information after the data source information of the data source is updated, and then update the metadata information of the same data source in the database, which can effectively ensure the consistency of the metadata information stored in the database with the metadata information corresponding to the corresponding data source.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a data processing method, including: obtaining data source information and first metadata information of multiple data sources, and storing the data source information and the first metadata information of each of the data sources in a database; in response to an update of the data source information of at least one of the data sources, obtaining second metadata information corresponding to the updated data source information; and replacing the first metadata information corresponding to the same data source in the database with the second metadata information.

[0006] In some embodiments, the multiple data sources include multiple internal data sources and multiple external data sources; obtaining the data source information and first metadata information of the multiple data sources includes: using a first interface to obtain the data source information and first metadata information corresponding to the multiple internal data sources and the multiple external data sources; and then storing the data source information and the corresponding first metadata information in the database in a distributed manner.

[0007] In some embodiments, it also includes: using a second interface to maintain the data source information and the first metadata information stored in the database; wherein the first interface is used to interface with a computing engine to obtain the data source information and the first metadata information; and the second interface is used to interface with a management interface to maintain the data source information and the first metadata information.

[0008] In some embodiments, in response to an update of data source information of at least one of the data sources, obtaining second metadata information corresponding to the updated data source information includes: in response to an update of the data source information of the data source, obtaining second metadata information corresponding to the updated data source information by using active acquisition and / or passive acquisition methods.

[0009] In some embodiments, the multiple data sources include multiple internal data sources; in response to the update of data source information of the internal data source, metadata change information of the external data source is obtained from the message queue using a passive acquisition method; and the second metadata information corresponding to the updated data source information is obtained based on the metadata change information.

[0010] In some embodiments, the multiple data sources also include multiple external data sources; in response to the update of data source information of the external data source, metadata acquisition is performed on the updated data source information using an active acquisition method to obtain the corresponding second metadata information.

[0011] In some embodiments, it also includes: obtaining data source information of the data source, logging into the data source using the data source information, and determining the update status of all tables of the data source; when there is at least one table that has not been updated, obtaining table structure information; when the table structure information is not empty, generating a structure code in an encrypted manner, determining whether the table structure information has changed based on the change of the structure code, and then determining the update status of the data source information of the data source.

[0012] In some embodiments, the use of the second metadata information to replace the first metadata information corresponding to the same data source in the database includes: after the metadata information of at least one of the data sources is updated, deleting the first metadata information corresponding to the updated data source information in the database; and inserting the second metadata information into the database, wherein the second metadata information corresponds to the updated data source information of the same data source.

[0013] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, the electronic device comprising a memory and a processor coupled to the memory, the memory storing at least one computer program, and the at least one computer program, when loaded and executed by the processor, is used to implement the above data processing method.

[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, wherein the computer-readable storage medium has at least one program, and when the at least one program is loaded and executed by the processor, it is used to implement the above data processing method.

[0015] Different from the current technology, the data processing method provided by the present application includes: obtaining data source information and first metadata information of multiple data sources, and storing the data source information and first metadata information of each data source in a database; in response to the update of the data source information of at least one data source, obtaining the second metadata information corresponding to the updated data source information; using the second metadata information to replace the first metadata information corresponding to the same data source in the database; that is, in the present application, the data source information and first metadata information of multiple data sources are stored in the database, and after the data source information of the data source is updated, the second metadata information corresponding to the updated data source information is found and obtained, and then the first metadata information stored in the database is replaced with the second metadata information corresponding to the same data source to complete the update of the metadata information, thereby ensuring the consistency of the metadata information stored in the database and the metadata information corresponding to the data source, facilitating data management and analysis, and thereby improving data query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1 It is a flowchart of an embodiment of a data processing method in this application;

[0018] Figure 2 It is a structural diagram of an embodiment of a metadata management service system in this application;

[0019] Figure 3 It is a flowchart of an embodiment of updating metadata information in this application;

[0020] Figure 4 It is a structural diagram of an embodiment of a data processing system in the present application;

[0021] Figure 5 It is a structural schematic diagram of an embodiment of an electronic device in the present application;

[0022] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium in the present application. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0024] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0025] Traditional database query systems target the big data analysis needs of their respective industries, and each system has an independent metadata management service. Therefore, the metadata management services between multiple database query systems cannot be well integrated. When using the computing engine to query across data sources, the data source needs to be manually configured, and unified management cannot be achieved. This is inconvenient to use, affecting data management and analysis, and in turn affecting data query efficiency.

[0026] Therefore, a data processing method is provided, which stores data source information and first metadata information of multiple data sources in a database, and after the data source information of the data source is updated, finds and obtains second metadata information corresponding to the updated data source information, and then replaces the first metadata information stored in the database with the second metadata information corresponding to the same data source to complete the update of the metadata information, thereby ensuring the consistency of the metadata information stored in the database and the metadata information corresponding to the data source, facilitating data management and analysis, and thereby improving data query efficiency.

[0027] See also Figure 1 , Figure 1 is a flow chart of an embodiment of a data processing method in the present application; it should be noted that if there are substantial results, the method of the present application is not limited to Figure 1 The process sequence shown is limited.

[0028] like Figure 1 As shown, the data processing method includes the following steps.

[0029] S10: Acquire data source information and first metadata information of multiple data sources, and store the data source information and first metadata information of each data source in a database.

[0030] Among them, data source refers to the source of data, such as the existing data storage structure. Because different industries have different data sources, there will be multiple data sources; data source information refers to the relevant information of the data stored in the data source, such as address, login name, password, repository information, data, etc.; metadata information refers to the data that describes the data. Metadata is defined by the data fields and data types in the data source. For example, metadata can describe the table structure, column information, relationships and constraints in the data source; that is, the code and documents that describe the data, including data name, field, size, format, category, data type, data range, as well as data source, processing method and other elements. Metadata reveals the essential characteristics of data and can provide support for various applications to manage, integrate and extract data; metadata is the basis of data integration.

[0031] Specifically, for multiple different data sources, the data source information and the corresponding first metadata information of each data source are obtained respectively, and the data source information and the corresponding first metadata information of each data source are uniformly stored in a database, where the database is a database that stores data source information and metadata information.

[0032] S20. In response to update of data source information of at least one data source, obtain second metadata information corresponding to the updated data source information.

[0033] The data source information of the data source is updated, which means that the data in the data source is updated. Therefore, the metadata information corresponding to the data source will also be updated accordingly. Therefore, new metadata information needs to be obtained.

[0034] Specifically, because the data within the data source may be constantly changing and updated, it is necessary to respond to the update of the data source information of the data source in a timely manner; and because there are multiple data sources, there may be situations where the data source information of multiple data sources is updated. Therefore, it is necessary to respond to the update of the data source information of at least one data source, and then obtain metadata information of the data source information of the updated data source, that is, obtain the second metadata information corresponding to the updated data source information.

[0035] S30: Use the second metadata information to replace the first metadata information corresponding to the same data source in the database.

[0036] Among them, the first metadata information is already stored in the database, and the second metadata information is obtained from the data source information after the data source is updated. Therefore, the second metadata information is the latest. In order to ensure the consistency of the metadata information, the first metadata information needs to be replaced with the second metadata information.

[0037] Specifically, after obtaining the second metadata information corresponding to the data source information of the updated data source, the first metadata information of the same data source is obtained, and then the second metadata information is used to replace the first metadata information corresponding to the same data source in the database to complete the update of the metadata information in the database, so that the metadata information in the database remains consistent with the metadata information corresponding to the data source information at the data source end.

[0038] In this embodiment, for different data sources, the data source information and first metadata information of multiple data sources are stored in a database, and after the data source information of the data source is updated, the second metadata information corresponding to the updated data source information is discovered and obtained, and then the first metadata information stored in the database is replaced with the second metadata information corresponding to the same data source to complete the update of the metadata information, thereby ensuring the consistency of the metadata information stored in the database and the metadata information corresponding to the data source, facilitating data management and analysis, and thereby improving data query efficiency.

[0039] In some embodiments, the metadata management service system of metadata information may include a metadata management interface layer, a metadata management storage layer, and a metadata discovery layer.

[0040] See also Figure 2 , Figure 2 It is a structural diagram of an embodiment of a metadata management service system in this application.

[0041] like Figure 2 As shown, the metadata management service system 300 includes a metadata management interface layer 310 , a metadata management storage layer 320 , and a metadata discovery layer 330 .

[0042] Among them, the metadata management interface layer 310 includes a first interface and a second interface; the first interface is an acquisition interface, which is used to obtain data source information and corresponding first metadata information of an internal data source and an external data source. For example, the first interface is an RPC interface (Remote Procedure Call Protocol); the first interface is connected to the computing engine to obtain corresponding data source information and first metadata information from the internal data source and the external data source.

[0043] The second interface is a management interface, which is used to maintain the data source information and metadata information of the data source, such as adding, deleting, checking and modifying. For example, the second interface is a Rest API interface (Representational State Transfer Application Programming Interface). The second interface is connected to the management interface, that is, the web management interface is provided with a second interface for managing the data source information and metadata information of the data source. The second interface can be used to maintain the data source information and the corresponding first metadata information stored in the database, such as adding, deleting, erasing and modifying.

[0044] The metadata management storage layer 320 is used to store the data source information and corresponding metadata information of the data source, and is divided into external data sources and internal data sources. The internal data source refers to the data source within the big data cluster in the data center, such as HMS (Hive MetaStore, which is the metadata management service of Hive), Doris (distributed SQL database) and HBase (distributed, column-oriented open source database); the external data source refers to the data source outside the big data cluster, such as ES (Elastic Search, a document database), MySQL (relational database management system) and Oracle (OracleDatabase, a relational database management system), etc.; through distributed storage, it supports the storage and management of massive metadata information.

[0045] The metadata discovery layer 330 is used to automatically discover metadata information of internal data sources and external data sources, and update the metadata information to ensure the consistency of the stored metadata information and the metadata information on the data source end; and the metadata discovery layer supports update strategies such as scheduled update, message subscription, and active update.

[0046] In some embodiments, the multiple data sources may include at least one internal data source and at least one external data source, and step S10 of acquiring data source information and first metadata information of the multiple data sources may include the following operation process.

[0047] The data source information and the first metadata information corresponding to at least one internal data source and at least one external data source are obtained using the first interface, wherein the first interface is used to interface with the computing engine to execute the acquisition of the data source information and the first metadata information.

[0048] Among them, the first interface is an acquisition interface, which is used to obtain the data source information and the corresponding first metadata information of the internal data source and the external data source. For example, the first interface is an RPC interface (Remote Procedure Call Protocol); the internal data source refers to the data source within the big data cluster in the data center, such as HMS (Hive MetaStore, which is the metadata management service of Hive), Doris (distributed SQL database) and HBase (distributed, column-oriented open source database); the external data source refers to the data source outside the big data cluster, such as ES (Elastic Search, document database), MySQL (relational database management system) and Oracle (Oracle Database, relational database management system); the data center is a platform that integrates multiple aspects such as data management, data governance, and data services; the computing engines (Spark / Flink / Doris) are all connected to the interface of the metadata management service system to realize the management and access of metadata.

[0049] Specifically, the first interface is used to connect to the computing engine. Therefore, the first interface can be used to connect to at least one external data source and at least one internal data source, and then the data source information and the corresponding first metadata information of the at least one external data source and the at least one internal data source are obtained through the first interface, that is, the data source information and the corresponding first metadata information of each data source are obtained.

[0050] Then, each data source information and the corresponding first metadata information are stored in a database in a distributed manner.

[0051] Among them, the database is the database corresponding to the data middle platform, a platform for data management, data governance, data services and other aspects.

[0052] Specifically, after obtaining the data source information and the corresponding first metadata information of each data source, the data source information and the corresponding first metadata information of the same data source are stored together according to the different data sources, while the data source information and the corresponding first metadata information of different data sources are stored in a distributed manner.

[0053] In some embodiments, the second interface may also be used to maintain the data source information and the first metadata information stored in the database; wherein the second interface is used to interface with the management interface to perform maintenance on the data source information and the first metadata information.

[0054] Among them, the second interface is a management interface, which is used to maintain the data source information and metadata information of the data source, such as adding, deleting, checking and modifying; for example, the second interface is a Rest API interface (Representational State Transfer Application Programming Interface, an application programming interface for transferring the presentation layer state).

[0055] Specifically, the second interface is connected to the management interface, that is, the second interface for managing the data source information and metadata information of the data source is provided to the web management interface, and then the second interface can be used to maintain the data source information and the corresponding first metadata information stored in the database, such as performing maintenance operations such as adding, deleting, erasing and modifying.

[0056] In some embodiments, step S20, in response to the update of data source information of at least one data source, obtains second metadata information corresponding to the updated data source information, which may include the following operation flow.

[0057] In response to the update of data source information of at least one data source, second metadata information corresponding to the updated data source information is acquired by actively acquiring and / or passively acquiring.

[0058] Among them, because there are multiple data sources, one of the data sources may update the data source information, or multiple data sources may update the data source information. Therefore, the second metadata information will be obtained corresponding to the update of the data source information of each data source; because the metadata information corresponding to the data source information of the data source is the latest, therefore, after the data source information is updated, the corresponding metadata information is also updated, that is, after the data source information is updated, the corresponding second metadata information exists.

[0059] Specifically, after the data source information of at least one data source is updated, in response to the update of the data source information, the second metadata information corresponding to the updated data source information is obtained through the first interface using an active acquisition method, or the second metadata information corresponding to the updated data source information is obtained using a passive acquisition method.

[0060] Further, the multiple data sources include at least one internal data source.

[0061] In response to the update of the data source information of the internal data source, the metadata change information of the internal data source can be obtained from the message queue by using a passive acquisition method.

[0062] The passive acquisition method is to passively receive update information of the data source information, such as subscribing to messages.

[0063] Specifically, as long as there is an update to the data source information of the internal data source, the metadata change information corresponding to the internal data source is subscribed to through the message queue, and the internal data source actively pushes the metadata change information to the message queue, so that the metadata discovery service can obtain the metadata change information in real time by subscribing to the messages in the message queue, that is, the changes in the metadata information can be obtained in real time.

[0064] Furthermore, second metadata information corresponding to the updated data source information is obtained according to the metadata change information.

[0065] Specifically, if there is metadata change information, it means that the metadata information corresponding to the data source has changed. Therefore, the second metadata information corresponding to the updated data source information can be obtained according to the metadata change information.

[0066] Furthermore, the multiple data sources include at least one external data source.

[0067] In response to the update of the data source information of the external data source, metadata acquisition is performed on the updated data source information in an active acquisition manner to acquire corresponding second metadata information.

[0068] The active acquisition method is to actively search for update information of the data source information, for example, by periodic acquisition or active triggered acquisition.

[0069] Specifically, as long as there is an update to the data source information of the external data source, the update information of the data source information is periodically acquired or actively triggered, and then metadata is acquired for the updated data source information to obtain the second metadata information corresponding to the updated data source information.

[0070] In this embodiment, by actively acquiring or passively acquiring, timely discovery and updating of metadata information can be ensured, providing a basis for consistency of subsequent metadata information.

[0071] Furthermore, it also includes determining the update status of the data source information of the data source through the change of the table structure information, including the following operation steps.

[0072] Get the data source information of the data source, log in to the data source using the data source information, and determine the update status of all tables of the data source.

[0073] The data source information includes the address, login name, password, library information and other related data of the data source. Therefore, the data source information can be used to log in to the data source.

[0074] Specifically, after obtaining the data source information of the data source, the data source is logged in using the login name and password in the data source information to obtain the table information of all tables in the data source and the update status of the tables.

[0075] When at least one table has not been updated, obtain the table structure information.

[0076] When at least one table has not been updated, it means that some tables in the data source have not been updated; therefore, it is necessary to obtain the table structure information in the data source.

[0077] When the table structure information is not empty, a structure code is generated in an encrypted manner, and whether the table structure information has changed is determined by the change of the structure code, thereby determining the update status of the data source information of the data source.

[0078] The encryption method may be a hash encryption method, such as using a Hash / MD5 encryption algorithm for calculation.

[0079] Specifically, it is determined whether the table structure information is empty, and when the table structure information is not empty, a corresponding structure code is generated in an encryption manner, and then it is determined whether the table structure information has changed based on the structure codes calculated at least twice.

[0080] For example, if the structural codes of the two calculations are different, it means that the table structure information has changed, and then it is determined that the data source information of the data source has changed, and the update status of the data source information of the data source is determined based on the change, and then the table structure information in the database is deleted, and the updated table structure information is inserted.

[0081] If the schema codes calculated twice are the same, the table structure information has not changed, and then check again whether all tables under the data source have been updated. If all tables under the data source have been updated, the update process ends.

[0082] In some embodiments, if the table structure information is empty, it means that the table with empty table structure information has been deleted in the data source. Therefore, it is necessary to delete the table with empty table structure information and related data in the database as well; and then check again whether all tables under the data source have been updated.

[0083] In this embodiment, an encoding method is used to encode the table structure information and fields of the data source using an encryption algorithm for comparison. Only one comparison is required to determine whether the metadata has changed, which greatly improves the efficiency of metadata update comparison.

[0084] In some embodiments, step S30 uses the second metadata information to replace the first metadata information corresponding to the same data source in the database, which may include the following operation process.

[0085] After the metadata information of at least one data source is updated, the first metadata information corresponding to the updated data source information in the database is deleted.

[0086] Specifically, in order to free up database space, after the metadata information corresponding to the data source is updated, that is, after the second metadata information of the updated data source is obtained, the first metadata information corresponding to the updated data source information originally stored in the database may be deleted.

[0087] Then, the second metadata information is inserted into the database, wherein the second metadata information corresponds to the updated data source information of the same data source.

[0088] Specifically, before or after deleting the first metadata information, the second metadata information corresponding to the same data source may be inserted into the database, and the second metadata information may correspond to the updated data source information of the same data source.

[0089] For the metadata information update principle, please refer to the attached figure.

[0090] See also Figure 3 , Figure 3 It is a flowchart of an embodiment of updating metadata information in this application.

[0091] like Figure 3 As shown, 1. First, obtain the data source information of the data source, and use the data source information to log in to the data source; 2. Check the table structure information of all tables in the data source and the update status of the tables. When all tables have been updated, the metadata update is completed; 3. If there is at least one table that has not been updated, obtain the table structure information of the table; 4. Determine whether the table structure information is empty; 5. When the table structure information is not empty, generate a structure code in an encrypted manner; 6. Then, based on the structure code calculated at least twice, determine whether the table structure information has changed, and then determine the update status of the data source information of the data source; 7. If the structure codes calculated twice are different, it means that the table structure information has changed, then delete the table structure information in the database, and insert the updated table structure information, and then check again whether all tables under the data source have been updated; 8. If the table structure information is empty, it means that the table with empty table structure information has been deleted in the data source. Therefore, it is necessary to delete the table with empty table structure information and related data in the database; then check again whether all tables under the data source have been updated.

[0092] In this embodiment, data source information and first metadata information of multiple data sources are stored in a database, and after the data source information of a data source is updated, second metadata information corresponding to the updated data source information is discovered and acquired, and then the first metadata information stored in the database is replaced with the second metadata information corresponding to the same data source to complete the update of the metadata information, thereby ensuring the consistency of the metadata information stored in the database and the metadata information corresponding to the data source, facilitating data management and analysis, and thereby improving data query efficiency.

[0093] In some embodiments, a data processing system is also provided.

[0094] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of a data processing system in the present application.

[0095] like Figure 4 As shown, the data processing system 400 includes a first acquisition module 410, a second acquisition module 420 and an update module 430; wherein the first acquisition module 410 is used to acquire data source information and first metadata information of multiple data sources, and store the data source information and first metadata information of each data source in a database; the second acquisition module 420 responds to the update of the data source information of at least one data source, and acquires second metadata information corresponding to the updated data source information; the update module 430 replaces the first metadata information corresponding to the same data source in the database with the second metadata information.

[0096] In some embodiments, an electronic device is also provided.

[0097] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of an electronic device in an embodiment of the present application. The electronic device can execute the data processing steps in the above method.

[0098] The electronic device 500 includes a memory 520, a processor 510 coupled to the memory, and at least one computer program stored in the memory 520 and executable on the processor 510; when the processor 510 loads and executes the at least one computer program, it is used to implement the data processing steps in the above method. For related content, please refer to the detailed description in the above method, which will not be repeated here.

[0099] In some embodiments, a computer-readable storage medium is also included.

[0100] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium in the present application.

[0101] The computer readable storage medium 600 stores at least one program 610, which is used to implement the data processing steps in the above method when loaded and executed by the processor. For related contents, please refer to the detailed description of the above method, which will not be repeated here.

[0102] The above scheme stores the data source information and the first metadata information of multiple data sources in a database, and after the data source information of the data source is updated, finds and obtains the second metadata information corresponding to the updated data source information, and then replaces the first metadata information stored in the database with the second metadata information corresponding to the same data source to complete the update of the metadata information, thereby ensuring the consistency of the metadata information stored in the database and the metadata information corresponding to the data source, facilitating data management and analysis, and thereby improving data query efficiency.

[0103] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0107] The above description is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Using a first interface to obtain data source information and first metadata information of multiple data sources, and storing the data source information and the first metadata information of each of the data sources in a database, wherein the multiple data sources include at least one internal data source and at least one external data source, the internal data source is a data source within the big data cluster corresponding to the data center, and the external data source is a data source outside the big data cluster, and the first interface adopts a remote procedure call protocol; In response to the update of the data source information of the internal data source, the second metadata information corresponding to the updated data source information of the internal data source is obtained through the first interface in a passive acquisition manner; or in response to the update of the data source information of the external data source, the second metadata information corresponding to the updated data source information of the external data source is obtained through the first interface in an active acquisition manner; wherein, the data source information of the data source is obtained, and the update status of all tables in the data source is determined; the table structure information of the table that has not been updated in the data source is obtained; when the table structure information is not empty, a structure code is generated in an encrypted manner, and the change of the table structure information is determined by the change of the structure code, and then the update status of the data source information of the data source is determined; The first metadata information corresponding to the same data source in the database is replaced by the second metadata information through a second interface, wherein the second interface is a Rest API interface.

2. The data processing method according to claim 1, characterized in that: The obtaining of data source information and first metadata information of multiple data sources includes: Acquire data source information and first metadata information corresponding to at least one of the internal data sources and at least one of the external data sources using a first interface, wherein the first interface is used to interface with a computing engine to acquire the data source information and the first metadata information; Then, each of the data source information and the corresponding first metadata information are stored in the database in a distributed manner.

3. The data processing method according to claim 2, characterized in that: Also includes: Maintaining the data source information and the first metadata information stored in the database using the second interface; The second interface is used to connect to a management interface to perform maintenance on the data source information and the first metadata information.

4. The data processing method according to claim 1, characterized in that: In response to the data source information of the internal data source being updated, obtaining metadata change information of the internal data source from a message queue in a passive acquisition manner; The second metadata information corresponding to the updated data source information is obtained according to the metadata change information.

5. The data processing method according to claim 1, characterized in that: Also includes: Acquire data source information of the data source, log in to the data source using the data source information, and determine the update status of all tables of the data source; When there is at least one table that has not been updated, obtain the table structure information; When the table structure information is not empty, a structure code is generated in an encrypted manner, and whether the table structure information has changed is determined by a change in the structure code, thereby determining the update status of the data source information of the data source.

6. The data processing method according to claim 1, characterized in that: The using the second metadata information to replace the first metadata information corresponding to the same data source in the database includes: After the metadata information of at least one of the data sources is updated, deleting the first metadata information corresponding to the updated data source information in the database; The second metadata information is inserted into the database, wherein the second metadata information corresponds to the updated data source information of the same data source.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor coupled to the memory, the memory stores at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the data processing method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium has at least one program, and when the at least one program is loaded and executed by the processor, it is used to implement the data processing method according to any one of claims 1 to 6.

Images