A metadata inspection method, device, equipment and medium for a big data platform

Abstract

The present application discloses a metadata inspection method, device, equipment and medium for a big data platform. The method includes: when a communication connection is established with a metadata master library in a target big data platform, first operation data of the metadata master library is collected using test data; the operation status of the metadata master library is parsed using the first operation data to obtain a first parsing result; when a communication connection is established with a metadata backup library in a target big data platform, a target command is sent to the metadata backup library to collect second operation data of the metadata backup library using the target command; the operation status of the metadata backup library is parsed using the second operation data to obtain a second parsing result. Through this method, the metadata in the big data platform can be inspected at any time, and the operation status of the metadata master library and the metadata backup library in the big data platform can be grasped in real time, thereby significantly improving the user experience when using the big data platform.

Description

Technical Field

[0001] The present invention relates to the field of server technology, and in particular to a metadata inspection method, device, equipment and medium for a big data platform. Background Art

[0002] In the prior art, a big data platform is usually used to monitor and manage the massive amount of data in an enterprise. In order to improve the availability of the metadata database in the big data platform, the metadata database is generally set to a master-slave mode.

[0003] However, in this configuration mode, the big data platform cannot detect fault metadata in a timely manner or accurately locate the fault metadata, which greatly affects the user experience when using the big data platform. Currently, there is no effective solution to this technical problem. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a metadata inspection method, device, equipment and medium for a big data platform to improve the user experience when using the big data platform. The specific solution is as follows:

[0005] A metadata inspection method for a big data platform, applied to a metadata inspection tool, includes:

[0006] When a communication connection is established with a metadata master library in a target big data platform, first operating data of the metadata master library is collected using test data;

[0007] Analyzing the operating status of the metadata master library using the first operating data to obtain a first analysis result;

[0008] When a communication connection is established with the metadata standby repository in the target big data platform, a target command is sent to the metadata standby repository to collect second operating data of the metadata standby repository using the target command;

[0009] The operating status of the metadata standby repository is analyzed using the second operating data to obtain a second analysis result.

[0010] Preferably, the process of collecting the first operation data of the metadata master library using the test data includes:

[0011] Creating a test database in the metadata master repository, and creating a test table in the test database;

[0012] Target data is inserted into the test table, and the first operation data of the metadata master library is collected using the target data.

[0013] Preferably, after the process of collecting the first operating data of the metadata master library using the target data, the process further includes:

[0014] The test database and the test table are deleted from the metadata master repository, and the communication connection with the metadata master repository is disconnected.

[0015] Preferably, the process of parsing the operating status of the metadata master library using the first operating data to obtain a first parsing result includes:

[0016] If the first operation data is an operation success message returned by the target data, it is determined that the metadata master database is in a normal state;

[0017] If the first operation data is an operation failure message returned by the target data, it is determined that the metadata master library is in an abnormal state, and the abnormal location of the metadata master library is determined using the operation failure message.

[0018] Preferably, the target big data platform is specifically Ambari, the metadata master database is specifically a MySQL master database, and the metadata standby database is specifically a MySQL standby database.

[0019] Preferably, when a communication connection is established with the metadata standby repository in the target big data platform, a target command is sent to the metadata standby repository to collect the second operating data of the metadata standby repository using the target command, including:

[0020] When a communication connection is established with the MySQL standby database in the Ambari, a show salve status command is sent to the MySQL standby database, so as to collect the second operating data of the MySQL standby database using the show salve status command.

[0021] Preferably, it also includes:

[0022] Push the first parsing result and the second parsing result to a target user.

[0023] Accordingly, the present invention also discloses a metadata inspection device for a big data platform, comprising:

[0024] A first data acquisition module is configured to collect first operating data of the metadata master library using test data when a communication connection is established with the metadata master library in the target big data platform;

[0025] a first result parsing module, configured to parse the operating status of the metadata master library using the first operating data to obtain a first parsing result;

[0026] a second data acquisition module, configured to, when establishing a communication connection with the metadata standby repository in the target big data platform, send a target command to the metadata standby repository, so as to collect second operating data of the metadata standby repository using the target command;

[0027] The second result parsing module is configured to parse the operating status of the metadata standby database using the second operating data to obtain a second parsing result.

[0028] Accordingly, the present invention also discloses a metadata inspection device for a big data platform, comprising:

[0029] memory for storing computer programs;

[0030] The processor is used to implement the steps of the metadata inspection method of a big data platform disclosed above when executing the computer program.

[0031] Correspondingly, the present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the metadata inspection method of a big data platform disclosed above are implemented.

[0032] It can be seen that in the metadata inspection method provided by the present invention, when the metadata inspection tool establishes a communication connection with the metadata master library in the target big data platform, the first operation data of the metadata master library is collected using test data, and the operation status of the metadata master library is analyzed using the first operation data to obtain a first analysis result; when the metadata inspection tool establishes a communication connection with the metadata backup library in the target big data platform, the metadata inspection tool is used to send a target command to the metadata backup library to use the target command to collect the second operation data of the metadata backup library. When the second operation data of the metadata backup library is collected, the operation status of the metadata backup library is analyzed using the second operation data to obtain a second analysis result. Obviously, compared with the existing technology, through such a setting, the metadata in the big data platform can be inspected at any time and the operation status of the metadata master library and metadata backup library in the big data platform can be grasped in real time. In this way, the big data platform can promptly detect the faulty metadata that occurs within it and accurately locate the faulty metadata, thereby significantly improving the user experience when using the big data platform. Accordingly, the metadata inspection device, equipment and medium for a big data platform provided by the present invention also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0034] Figure 1 A flowchart of a metadata inspection method for a big data platform provided by an embodiment of the present invention;

[0035] Figure 2 A structural diagram of a metadata inspection device for a big data platform provided by an embodiment of the present invention;

[0036] Figure 3 This is a structural diagram of a metadata inspection device for a big data platform provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] See Figure 1 , Figure 1 This is a flowchart of a metadata inspection method for a big data platform provided by an embodiment of the present invention. The metadata inspection method includes:

[0039] Step S11: When a communication connection is established with the metadata master database in the target big data platform, first operating data of the metadata master database is collected using test data;

[0040] Step S12: Analyze the operating status of the metadata master library using the first operating data to obtain a first analysis result;

[0041] Step S13: When a communication connection is established with the metadata standby repository in the target big data platform, a target command is sent to the metadata standby repository to collect second operating data of the metadata standby repository using the target command;

[0042] Step S14: Analyze the operating status of the metadata standby database using the second operating data to obtain a second analysis result.

[0043] In this embodiment, a metadata inspection method for a big data platform is provided. By utilizing this metadata inspection method, the big data platform can promptly detect faulty metadata occurring within it and accurately locate the faulty metadata, thereby significantly improving the user experience when using the big data platform.

[0044] In this metadata inspection method, a metadata inspection tool is used as the execution subject for specific description. When it is necessary to inspect the metadata in the big data platform, the metadata inspection tool will first establish a communication connection with the metadata master library in the target big data platform. After the metadata inspection tool establishes a communication connection with the metadata master library in the big data platform, the metadata inspection tool will insert test data into the metadata master library and collect the first operating data of the metadata master library based on the feedback information returned by the test data. It can be understood that after the metadata inspection tool collects the first operating data, because the first operating data can represent the operating status of the metadata master library, the metadata inspection tool can determine the operating status of the metadata master library based on the first analytical result after parsing the first operating data of the metadata master library.

[0045] After the metadata inspection tool has completed the inspection of the metadata master library in the target big data platform, it can then inspect the metadata standby library in the target big data platform. Specifically, the metadata inspection tool will first establish a communication connection with the metadata standby library in the target big data platform. After the metadata inspection tool establishes a communication connection with the metadata standby library, the metadata inspection tool will send a target command to the metadata standby library; after the metadata standby library receives the target command sent by the metadata inspection tool, the metadata standby library will send the second operating data of the metadata standby library to the metadata inspection tool. It can be understood that because the second operating data of the metadata standby library can represent the operating status of the metadata standby library, the metadata inspection tool can determine the operating status of the metadata standby library based on the second analytical result after parsing the second operating data and obtaining the second analytical result.

[0046] The metadata inspection tool provided in this embodiment allows for real-time inspection of metadata within the big data platform, providing real-time visibility into the operational status of the platform's primary and backup metadata repositories. This configuration enables the big data platform to promptly detect and accurately locate faulty metadata, significantly improving the user experience when using the platform.

[0047] It can be seen that in the metadata inspection method provided in this embodiment, when the metadata inspection tool establishes a communication connection with the metadata master library in the target big data platform, the first operation data of the metadata master library is collected using test data, and the operation status of the metadata master library is analyzed using the first operation data to obtain a first analysis result; when the metadata inspection tool establishes a communication connection with the metadata backup library in the target big data platform, the metadata inspection tool is used to send a target command to the metadata backup library to collect the second operation data of the metadata backup library using the target command. When the second operation data of the metadata backup library is collected, the operation status of the metadata backup library is analyzed using the second operation data to obtain a second analysis result. Obviously, compared with the existing technology, through such a setting method, the metadata in the big data platform can be inspected at any time and the operation status of the metadata master library and metadata backup library in the big data platform can be grasped in real time. In this way, the big data platform can promptly detect the faulty metadata that occurs within it and accurately locate the faulty metadata, thereby significantly improving the user experience when using the big data platform.

[0048] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. As a preferred implementation, the above step of collecting the first running data of the metadata master library using the test data includes:

[0049] Create a test database in the metadata master repository and create test tables in the test database.

[0050] Insert target data into the test table, and use the target data to collect the first running data of the metadata master library.

[0051] In this embodiment, when using test data to collect the first operational data from the metadata master repository, it is first necessary to create a test database in the metadata master repository and a test table within the test database. Next, the target data is inserted into the test table. It is conceivable that after inserting the target data into the test table, the metadata master repository will return feedback data corresponding to the target data to the metadata inspection tool. When the metadata inspection tool receives the feedback data from the metadata master repository, it is equivalent to having collected the first operational data from the metadata master repository.

[0052] As a preferred embodiment, after the above step of collecting the first operation data of the metadata master library using the target data, the following steps are further included:

[0053] Delete the test database and test tables from the metadata master repository and disconnect the communication with the metadata master repository.

[0054] Specifically, in this embodiment, after the first operational data of the metadata master database is collected using the target data, the test database and test tables can be deleted from the metadata master database, and the communication connection with the metadata master database can be disconnected. It is conceivable that this configuration can relatively reduce the resource usage of the metadata master database.

[0055] As a preferred embodiment, the above step of parsing the operating status of the metadata master library using the first operating data to obtain the first parsing result includes:

[0056] If the first operation data is an operation success message returned by the target data, it is determined that the metadata master database is in a normal state;

[0057] If the first operation data is an operation failure message returned by the target data, it is determined that the metadata master library is in an abnormal state, and the operation failure message is used to determine the abnormal location of the metadata master library.

[0058] It is understandable that after inserting target data into the test table, the metadata master library will definitely return corresponding feedback data to the metadata inspection tool, wherein the feedback data is the first running data collected by the metadata inspection tool from the metadata master library.

[0059] Because the first operation data contains the current operation information of each metadata in the metadata master library, the current operation status of the metadata master library can be learned by parsing the first operation data. Specifically, if the first operation data collected by the metadata inspection tool is an operation success message returned by the target data, it means that the metadata master library is in a normal operating state, that is, all metadata in the metadata master library are in a normal state. If the first operation data collected by the metadata inspection tool is an operation failure message returned by the target data, it means that the metadata master library is in an abnormal state. At this time, the abnormal position in the metadata master library can be determined based on the operation failure message returned by the target data, that is, the specific position of the fault or abnormal metadata in the metadata master library can be determined based on the operation failure message returned by the target data.

[0060] Obviously, through the technical solution provided by this embodiment, the purpose of inspecting the metadata master library in the target big data platform can be achieved.

[0061] Based on the above embodiments, this embodiment further illustrates and optimizes the technical solution. As a preferred implementation method, the target big data platform is specifically Ambari, the metadata master database is specifically the MySQL master database, and the metadata standby database is specifically the MySQL standby database.

[0062] In practice, Ambari has become an effective management tool for distributed clusters because it supports the provisioning, management, and monitoring of Apache Hadoop clusters. Therefore, in this embodiment, the target big data platform is set to Ambari, which further improves the universality of the target big data platform in actual use.

[0063] In addition, because MySQL's master and standby databases can not only perform data backup, but also have the advantages of load balancing, balanced data distribution, high availability and high fault tolerance, when the metadata master and metadata standby databases in Ambari are set as MySQL master and MySQL standby databases, the reliability and convenience of the metadata master and metadata standby databases during use can be relatively improved.

[0064] As a preferred embodiment, the above step: when a communication connection is established with the metadata standby repository in the target big data platform, a target command is sent to the metadata standby repository to collect the second operating data of the metadata standby repository using the target command, includes:

[0065] When a communication connection is established with the MySQL standby database in Ambari, a showsalve status command is sent to the MySQL standby database to collect the second running data of the MySQL standby database using the show salve status command.

[0066] Specifically, in actual applications, when the target big data platform is set to Ambari, and the metadata master database and metadata standby database are set to the MySQL master database and MySQL standby database respectively, the metadata inspection tool first establishes a communication connection with the MySQL standby database when collecting the operating data of the MySQL standby database. After the metadata inspection tool establishes a communication connection with the MySQL standby database in Ambari, the metadata inspection tool can receive the second operating data fed back by the MySQL standby database to the metadata inspection tool by sending a showsalve status command to the MySQL standby database.

[0067] When the metadata inspection tool sends the show salve status command to the MySQL standby database, Slave_IO_Running indicates that the metadata inspection tool is connected to the metadata standby database, reading its logs locally and generating local log files. Slave_SQL_Running indicates that the local log files are read and the SQL commands in the log files are executed. Last_IO_Error indicates errors and related information that occurred during the IO phase of synchronization between the MySQL master and standby databases.

[0068] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. As a preferred implementation, the metadata inspection method further includes:

[0069] Push the first parsing result and the second parsing result to the target user.

[0070] In this embodiment, in order to enable users to clearly know the current operating status of the metadata master library and the metadata backup library in the target big data platform, after the metadata inspection tool parses and obtains the first analysis result of the metadata master library and the second analysis result of the metadata backup library, the metadata inspection tool can also actively push the first analysis result and the second analysis result to the target user.

[0071] It can be imagined that when the metadata inspection tool pushes the analysis results of the metadata master library and the metadata backup library to the target user, the target user can not only know the real-time operating status of the metadata master library and the metadata backup library in the target big data platform in a timely manner, but also, based on the analysis results of the metadata master library and the metadata backup library, can accurately locate the metadata location where errors or failures occur in the metadata master library and / or the metadata backup library, which greatly increases the ease of use for users when using the target big data platform.

[0072] Obviously, the technical solution provided by this embodiment can further improve people's user experience when using the target big data platform.

[0073] See Figure 2 , Figure 2 This is a structural diagram of a metadata inspection device for a big data platform provided by an embodiment of the present invention, the metadata inspection device comprising:

[0074] The first data acquisition module 21 is configured to collect first operating data of the metadata master library using test data when establishing a communication connection with the metadata master library in the target big data platform;

[0075] A first result parsing module 22 is configured to parse the operating status of the metadata master library using the first operating data to obtain a first parsing result;

[0076] The second data acquisition module 23 is configured to, when a communication connection is established with the metadata standby repository in the target big data platform, send a target command to the metadata standby repository to collect second operating data of the metadata standby repository using the target command;

[0077] The second result analysis module 24 is configured to analyze the operation status of the metadata standby database using the second operation data to obtain a second analysis result.

[0078] A metadata inspection device for a big data platform provided in an embodiment of the present invention has the beneficial effects of the metadata inspection method for a big data platform disclosed above.

[0079] See Figure 3 , Figure 3 This is a structural diagram of a metadata inspection device for a big data platform provided by an embodiment of the present invention. The metadata inspection device includes:

[0080] Memory 31, for storing computer programs;

[0081] The processor 32 is configured to implement the steps of the metadata inspection method for a big data platform disclosed above when executing a computer program.

[0082] The metadata inspection device for a big data platform provided by an embodiment of the present invention has the beneficial effects of the metadata inspection method for a big data platform disclosed above.

[0083] Correspondingly, an embodiment of the present invention further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the metadata inspection method of a big data platform disclosed above are implemented.

[0084] A computer-readable storage medium provided by an embodiment of the present invention has the beneficial effects of the aforementioned metadata inspection method for a big data platform.

[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0086] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0087] The above is a detailed introduction to the metadata inspection method, device, equipment and medium of a big data platform provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A metadata inspection method for a big data platform, characterized in that: Applicable to metadata inspection tools, including: When a communication connection is established with a metadata master library in the target big data platform, first operation data of the metadata master library is collected using the test data; Analyzing the operation status of the metadata master library using the first operation data to obtain a first analysis result; When a communication connection is established with the metadata standby repository in the target big data platform, a target command is sent to the metadata standby repository to collect second operation data of the metadata standby repository using the target command; Analyzing the operation status of the metadata standby database using the second operation data to obtain a second analysis result; The process of collecting the first operation data of the metadata master library by using the test data includes: Creating a test database in the metadata master repository, and creating a test table in the test database; Inserting target data into the test table, and using the target data to collect the first operation data of the metadata master library; The process of parsing the operating state of the metadata master library using the first operating data to obtain a first parsing result includes: If the first operation data is an operation success message returned by the target data, it is determined that the metadata master library is in a normal state; If the first operation data is an operation failure message returned by the target data, it is determined that the metadata master library is in an abnormal state, and the abnormal position of the metadata master library is determined by using the operation failure message; The target big data platform is specifically Ambari, the metadata master database is specifically a MySQL master database, and the metadata standby database is specifically a MySQL standby database; When a communication connection is established with the metadata standby repository in the target big data platform, a target command is sent to the metadata standby repository to collect the second operation data of the metadata standby repository by using the target command, including: When a communication connection is established with the MySQL standby database in the Ambari, a show salve status command is sent to the MySQL standby database to collect the second operating data of the MySQL standby database using the show salve status command; wherein, when the metadata inspection tool sends the show salve status command to the MySQL standby database, Slave_IO_Running indicates that the metadata inspection tool is connected to the metadata standby database, and reads the log of the metadata standby database locally to generate a local log file; Slave_SQL_Running indicates reading the local log file and executing the SQL commands in the log file; Last_IO_Error indicates errors and related information that occurred in the IO stage when the MySQL master database and the MySQL standby database are synchronized.

2. The metadata inspection method according to claim 1, characterized in that: After the process of collecting the first operation data of the metadata master library using the target data, the method further includes: The test database and the test table are deleted from the metadata master repository, and the communication connection with the metadata master repository is disconnected.

3. The metadata inspection method according to any one of claims 1 to 2, characterized in that: Also includes: The first analysis result and the second analysis result are pushed to a target user.

4. A metadata inspection device for a big data platform, characterized in that: include: A first data acquisition module is used to collect first operation data of the metadata master library by using test data when establishing a communication connection with the metadata master library in the target big data platform; A first result analysis module, used to analyze the operation status of the metadata master library using the first operation data to obtain a first analysis result; A second data acquisition module is used to send a target command to the metadata standby repository when a communication connection is established with the metadata standby repository in the target big data platform, so as to collect second operation data of the metadata standby repository by using the target command; A second result analysis module, used to analyze the operation status of the metadata standby database using the second operation data to obtain a second analysis result; The first data acquisition module is specifically used to create a test database in the metadata master library and create a test table in the test database; insert target data into the test table and use the target data to collect the first operation data of the metadata master library; a first result parsing module, specifically configured to determine that the metadata master library is in a normal state if the first operation data is an operation success message returned by the target data; and determine that the metadata master library is in an abnormal state if the first operation data is an operation failure message returned by the target data, and determine the abnormal position of the metadata master library by using the operation failure message; The target big data platform is specifically Ambari, the metadata master database is specifically a MySQL master database, and the metadata standby database is specifically a MySQL standby database; The second data acquisition module is specifically used to send a show salve status command to the MySQL standby database when a communication connection is established with the MySQL standby database in the Ambari, so as to collect the second operating data of the MySQL standby database by using the show salve status command; wherein, when the metadata inspection tool sends the show salve status command to the MySQL standby database, Slave_IO_Running indicates that the metadata inspection tool is connected to the metadata standby database, and reads the log of the metadata standby database locally to generate a local log file; Slave_SQL_Running indicates reading the local log file and executing the SQL commands in the log file; Last_IO_Error indicates the error and related information that occurred in the IO stage when the MySQL master database and the MySQL standby database are synchronized.

5. A metadata inspection device for a big data platform, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of a metadata inspection method for a big data platform as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the metadata inspection method for a big data platform as described in any one of claims 1 to 3 are implemented.

Images