Metadata operation method and device of big data computing platform and computer equipment

Abstract

The invention relates to a metadata operation method and device of a big data computing platform and computer equipment. The method comprises the steps of establishing communication connection with a target computing cluster in response to an access operation of a client on a target data warehouse; the target data warehouse is used for storing metadata associated with the preset data processing component according to a preset metadata storage structure; and executing a metadata operation instruction input by the client for the target metadata object, determining storage position information corresponding to the target metadata object in the metadata storage structure, and controlling the target computing cluster to execute corresponding metadata operation on the target metadata object according to the storage position information. Obtaining a metadata operation result of the metadata operation instruction; and returning a metadata operation result of the metadata operation instruction to the client, and disconnecting the communication connection with the target computing cluster. By adopting the method, the metadata operation efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular, to a method, apparatus, computer device, computer-readable storage medium, and computer program product for metadata operation of a big data computing platform. Background Art

[0002] A big data computing platform refers to an integrated platform that integrates a variety of data processing tools and technologies for processing, storing, and analyzing large amounts of data. However, with the development of big data technology, the number of data processing components associated with the big data computing platform is increasing. Different data processing components often run independently on different computing clusters, which leads to the dispersion and duplication of the metadata of each data processing component. This greatly increases the maintenance complexity of the metadata of the data processing components and also results in the problem of inefficiency in data operations on the existing metadata. In addition, due to the lack of a unified metadata management system, cross-platform data sharing and interoperability become difficult, affecting the efficient operation of business systems based on the big data computing platform.

[0003] Therefore, there is a problem of low efficiency in metadata operation of the big data computing platform in the related art. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for metadata operation of a big data computing platform that can improve the efficiency of metadata operation of the big data computing platform in view of the above technical problems.

[0005] In a first aspect, this application provides a method for metadata operation of a big data computing platform, the method comprising:

[0006] In response to an access operation of a client to a target data warehouse, establish a communication connection with a target computing cluster; the target data warehouse is used to store the metadata associated with a preset data processing component according to a preset metadata storage structure; the preset data processing component includes at least two data processing components associated with a preset big data computing platform; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation;

[0007] Execute a metadata operation instruction input by the client for a target metadata object, determine the storage location information corresponding to the target metadata object in the metadata storage structure, and according to the storage location information, control the target computing cluster to perform a corresponding metadata operation on the target metadata object to obtain a metadata operation result of the metadata operation instruction;

[0008] Return the metadata operation result of the metadata operation instruction to the client, and disconnect the communication connection with the target computing cluster.

[0009] In one embodiment, when the user account logged in to the client does not have administrative privileges, the establishing a communication connection between the client and the target computing cluster in response to an access operation of the client to the target data warehouse includes:

[0010] In response to an access operation of the client to the target data warehouse, obtain a service access verification ticket submitted by the client; the service access verification ticket is a ticket for verifying the account identity of the user account generated according to a preset network authentication protocol;

[0011] Perform an identity authentication operation on the client according to the service access verification ticket;

[0012] In response to a verification passed event triggered by the identity authentication operation, establish a communication connection between the client and the target computing cluster.

[0013] In one embodiment, when the user account logged in to the client has administrative privileges, the method further includes:

[0014] In response to an opening operation of the client to the target data warehouse, enter a data warehouse configuration mode; the host in the data warehouse configuration mode is used to receive data warehouse configuration instructions; the data warehouse configuration instructions include at least one of an environment deployment instruction, a role grouping addition instruction, a computing cluster control instruction, an instance status control instruction, and an instance status monitoring instruction;

[0015] Execute the data warehouse configuration instruction, and trigger the target data warehouse to store the metadata associated with the preset data processing component according to the metadata storage structure.

[0016] In one embodiment, the method further includes:

[0017] Obtain data warehouse modeling information input for the big data computing platform; the data warehouse modeling information is used to represent the data warehouse hierarchical structure of the data warehouse model;

[0018] Construct the target data warehouse according to the data warehouse hierarchical structure represented by the data warehouse modeling information.

[0019] In one embodiment, the data warehouse model includes at least one of a data access layer, a data processing layer, or a data application layer;

[0020] The data access layer is used to access the original business data; the original business data includes at least one of the configuration metadata, system operation data, cluster operation data, storage metadata, computing job data, log data, or cluster operation data of the big data computing platform;

[0021] The data processing layer includes a data warehouse detail layer, a data warehouse summary layer, and a dimension common layer; the data warehouse detail layer is used to convert the original business data into a detail fact table according to the preset business logic; the data warehouse summary layer is used to summarize the detail data in the detail fact table according to the dimension information provided by the dimension common layer to obtain summary data;

[0022] The data application layer is used to provide at least one of a data insight function, a file merging function, a data middle platform function, an operation and maintenance monitoring function, a parameter recommendation function, or a data lake warehouse management function.

[0023] In one embodiment, the metadata storage structure includes metadata data domains obtained by dividing according to at least two preset data domain levels; the metadata data domains include at least one of a configuration domain, a cost domain, a storage domain, a computing domain, an operation domain, or a stability domain; each of the metadata data domains is associated with at least one sub-data domain.

[0024] In a second aspect, the present application also provides a metadata operation device for a big data computing platform, and the device includes:

[0025] An access module, which is used to establish a communication connection with a target computing cluster in response to an access operation of a client to a target data warehouse; the target data warehouse is used to store the metadata associated with a preset data processing component according to a preset metadata storage structure; the preset data processing component includes at least two data processing components associated with a preset big data computing platform; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation;

[0026] An execution module, which is used to execute the metadata operation instruction input by the client to a target metadata object, determine the storage location information of the target metadata object corresponding to the metadata storage structure, and according to the storage location information, control the target computing cluster to perform a corresponding metadata operation on the target metadata object to obtain the metadata operation result of the metadata operation instruction;

[0027] A return module, which is used to return the metadata operation result of the metadata operation instruction to the client and disconnect the communication connection with the target computing cluster.

[0028] In a third aspect, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0030] In a fifth aspect, the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0031] The above method, device, computer device, computer-readable storage medium, and computer program product for metadata operation of a big data computing platform establish a communication connection with a target computing cluster in response to an access operation of a client to a target data warehouse. The target data warehouse is used to store metadata of at least two data processing components associated with a preset big data computing platform according to a preset metadata storage structure; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation; execute the metadata operation instruction input by the client for the target metadata object, determine the storage location information corresponding to the target metadata object in the metadata storage structure, and according to the storage location information, control the target computing cluster to perform a corresponding metadata operation on the target metadata object to obtain the metadata operation result of the metadata operation instruction; return the metadata operation result of the metadata operation instruction to the client and disconnect the communication connection with the target computing cluster; in this way, it effectively realizes the effective docking of the client with different data processing components scattered in the big data platform, realizes the unified management of the metadata of different data processing components, avoids the data island phenomenon, improves the efficiency and consistency of metadata management, effectively reduces data duplicate storage and redundant calculation by simplifying cross-platform data management operations, thereby reducing the data interconnection cost, releasing computing, storage, and human resources, solving the metadata management problem existing in the current big data platform system, ensuring data access and operation consistency between different systems and platforms, improving the operability and reliability of data, and effectively improving the metadata operation efficiency of the big data computing platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for the description of the embodiments of the present application or related technologies. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0033] Figure 1 It is an application environment diagram of the metadata operation method in the big data computing platform in one embodiment;

[0034] Figure 2 It is a schematic flowchart of the metadata operation method in the big data computing platform in one embodiment;

[0035] Figure 3 It is a schematic diagram of a business model in one embodiment;

[0036] Figure 4 It is a schematic diagram of a metadata specification process in one embodiment;

[0037] Figure 5 It is a schematic diagram of a business process modeling model of a business model in one embodiment;

[0038] Figure 6 It is a schematic diagram of the layering of a metadata data warehouse in one embodiment;

[0039] Figure 7 It is a schematic diagram of the layering of another metadata data warehouse in one embodiment;

[0040] Figure 8 It is a schematic flowchart of the metadata operation method in the big data computing platform in another embodiment;

[0041] Figure 9 It is a structural block diagram of the metadata operation device in the big data computing platform in one embodiment;

[0042] Figure 10 It is an internal structure diagram of a computer device in one embodiment. Detailed implementation manners

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] When the relevant data collection and processing in this application book are applied in practice, the informed consent or separate consent of the personal information subject should be obtained strictly in accordance with the requirements of relevant national laws and regulations, and subsequent data use and processing behaviors should be carried out within the scope authorized by laws and regulations and the personal information subject.

[0045] A metadata operation method for a big data computing platform provided by an embodiment of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the client 102 communicates with the metadata management host 104 through the network. The data storage system can store the data that the metadata management host 104 needs to process. The data storage system can be integrated on the metadata management host 104, or can be placed on the cloud or other network servers.

[0046] In practical applications, in response to the client's access operation to the target data warehouse, the metadata management host 104 establishes a communication connection with the target computing cluster; the target data warehouse is used to store the metadata associated with the preset data processing components according to the preset metadata storage structure; the preset data processing components include at least two data processing components associated with the preset big data computing platform; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation; the metadata management host 104 executes the metadata operation instruction input by the client for the target metadata object, determines the storage location information corresponding to the target metadata object in the metadata storage structure, and according to the storage location information, controls the target computing cluster to perform the corresponding metadata operation on the target metadata object to obtain the metadata operation result of the metadata operation instruction; the metadata management host 104 returns the metadata operation result of the metadata operation instruction to the client and disconnects the communication connection with the target computing cluster.

[0047] Among them, the client 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The metadata management host 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0048] In an exemplary embodiment, as Figure 2 shown, a method for operating metadata of a big data computing platform is provided. Taking the metadata management host 104 in Figure 1 as an example for illustration, it includes the following steps S202 to step S206. Among them:

[0049] Step S202, in response to the client's access operation to the target data warehouse, establish a communication connection with the target computing cluster.

[0050] Among them, the target data warehouse is used to store the metadata associated with the preset data processing components according to the preset metadata storage structure.

[0051] Among them, the data warehouse, abbreviated as the data storehouse, is a system that stores a large amount of data from multiple different sources. After being sorted, cleaned, and transformed, it is used for business analysis and decision support.

[0052] In a specific implementation, as Figure 3 shown, when the client needs to perform a metadata operation on a target metadata object in the target computing cluster, the client can send an access request to the metadata management host to implement an access operation on the target data warehouse; the metadata management host can establish a communication connection with the target computing cluster in response to the access operation on the target data warehouse. Among them, the target computing cluster can include at least one of the Hdfs cluster, the Hive cluster, and the Yarn cluster.

[0053] In practical applications, as Figure 4 shown, to ensure the consistency and operability of metadata, the technical solution of the present disclosure also needs to define the metadata in a standardized manner, specifically including the following items:

[0054] Item 1: Table naming and field naming: Formulate a unified naming rule;

[0055] Item 2: Refresh period: Determine the update frequency of metadata, as shown in Table 1.

[0056]

[0057] Table 1

[0058] Item 3: Stock strategy: Define the data retention and cleaning strategy, as shown in Table 2.

[0059]

[0060] Table 2

[0061] Item 4: Granularity: Determine the refinement degree of data records.

[0062] Item 5: Statistical period: Set the period for data statistics and analysis, as shown in Table 3.

[0063]

[0064] Table 3

[0065] Item 6: Scheduling period: Determine the scheduling frequency of data processing tasks, as shown in Table 4.

[0066]

[0067] Table 4

[0068] Item 7: Partition Strategy: Define the partitioning method for large data tables, as shown in Table 5.

[0069]

[0070] Table 5

[0071] In addition, based on the metadata definition, standardize the definition of the indicator system, including atomic indicators and derived indicators:

[0072] Among them, the atomic indicator can refer to the basic indicator abstracted from the business process, with a clear business meaning. In practical applications, the atomic indicator can include action objects, business actions, and metrics.

[0073] Among them, the derived indicator is derived from the atomic indicator combined with the time period and modifiers, and is used for indicators in specific analysis scenarios. In practical applications, the derived indicator can include atomic indicators, time periods, and modifiers.

[0074] Among them, the preset data processing components include at least two data processing components associated with the preset big data computing platform.

[0075] Among them, the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation.

[0076] Step S204, execute the metadata operation instruction input by the client for the target metadata object, determine the storage location information of the target metadata object corresponding to the metadata storage structure, and according to the storage location information, control the target computing cluster to perform the corresponding metadata operation on the target metadata object to obtain the metadata operation result of the metadata operation instruction.

[0077] Among them, the metadata operation instruction can include a metadata addition instruction, a metadata deletion instruction, a metadata modification instruction, and a metadata query instruction.

[0078] In specific implementation, the metadata management host executes the metadata operation instruction input by the client for the target metadata object, determines the storage location information of the target metadata object corresponding to the metadata storage structure, and the metadata management host controls the target computing cluster to perform the corresponding metadata operation on the target metadata object according to the storage location information to obtain the metadata operation result of the metadata operation instruction.

[0079] For example, when the target computing cluster is a Hive cluster, the metadata management host can execute the metadata query instruction input by the client for the target metadata A in the Hive cluster. The metadata management host determines the storage location information corresponding to the target metadata A in the metadata storage structure. Specifically, the data directory A of the target metadata A; the metadata management host controls the Hive cluster to execute the query operation of the target metadata A according to the data directory A of the target metadata A, and obtains the metadata operation result of the metadata operation instruction, that is, the metadata content of the target metadata A is queried.

[0080] Step S206, return the metadata operation result of the metadata operation instruction to the client, and disconnect the communication connection with the target computing cluster.

[0081] In specific implementation, the metadata management host sends the metadata operation result of the data operation instruction to the client. Continuing with the above example, the metadata management host can send the metadata content of the queried target metadata A to the client; then, the metadata management host can disconnect the communication connection with the target computing cluster.

[0082] In the above metadata operation method of the big data computing platform, by responding to the access operation of the client to the target data warehouse, a communication connection with the target computing cluster is established. The target data warehouse is used to store the metadata of at least two data processing components associated with the preset big data computing platform according to the preset metadata storage structure; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation; execute the metadata operation instruction input by the client for the target metadata object, determine the storage location information corresponding to the target metadata object in the metadata storage structure, and according to the storage location information, control the target computing cluster to perform the corresponding metadata operation on the target metadata object, and obtain the metadata operation result of the metadata operation instruction; return the metadata operation result of the metadata operation instruction to the client, and disconnect the communication connection with the target computing cluster; in this way, it effectively realizes the effective docking of the client with different data processing components scattered in the big data platform, realizes the unified management of the metadata of different data processing components, avoids the data island phenomenon, improves the efficiency and consistency of metadata management, effectively reduces data duplicate storage and redundant calculations by simplifying cross-platform data management operations, thereby reducing the data interconnection cost, releasing computing, storage and human resources, solving the metadata management problems existing in the current big data platform system, ensuring data access and operation consistency between different systems and platforms, improving the operability and reliability of data, and effectively improving the metadata operation efficiency of the big data computing platform.

[0083] In an exemplary embodiment, when the user account logging in to the client does not have administrative privileges, in response to an access operation of the client to the target data warehouse, establishing a communication connection between the client and the target computing cluster includes: in response to the access operation of the client to the target data warehouse, obtaining a service access verification ticket submitted by the client; performing an authentication operation on the client according to the service access verification ticket; in response to a verification passed event triggered by the authentication operation, establishing a communication connection with the target computing cluster.

[0084] Wherein, the service access verification ticket is an account identity ticket generated according to a preset network authentication protocol for verifying the account identity of the user account.

[0085] In a specific implementation, when the user account logging in to the client does not have administrative privileges (that is, the user account logging in to the client is an ordinary user), in the process of the metadata management host establishing a communication connection between the client and the target computing cluster in response to the access operation of the client to the target data warehouse, it specifically includes: the metadata management host, in response to the access operation of the client to the target data warehouse, obtains the service access verification ticket submitted by the client. Specifically, when the metadata management host receives the access request of the client to the target data warehouse, the metadata management host obtains the service access verification ticket in the access request.

[0086] Then, the metadata management host performs an authentication operation on the client according to the service access verification ticket; the metadata management host, in response to a verification passed event triggered by the authentication operation, establishes a communication connection with the target computing cluster.

[0087] Wherein, the target computing cluster may include at least one of an Hdfs cluster, a Hive cluster, and a Yarn cluster.

[0088] The technical solution of this embodiment, by in response to the access operation of the client to the target data warehouse, obtaining the service access verification ticket submitted by the client, and performing an authentication operation on the client according to the service access verification ticket; in response to a verification passed event triggered by the authentication operation, establishing a communication connection with the target computing cluster, can thus achieve access control over the client and ensure the security of metadata operations.

[0089] In an exemplary embodiment, when the user account logging in to the client has administrative privileges, the method further includes: in response to an opening operation of the client to the target data warehouse, entering a data warehouse configuration mode; the host in the data warehouse configuration mode is used to receive data warehouse configuration instructions; executing the data warehouse configuration instructions, triggering the target data warehouse to store the metadata associated with the preset data processing components according to the metadata storage structure.

[0090] Among them, the data warehouse configuration instructions include at least one of environment deployment instructions, role grouping addition instructions, computing cluster control instructions, instance status control instructions, and instance status monitoring instructions.

[0091] Among them, the computing cluster control instructions include computing cluster addition instructions, computing cluster configuration instructions, or computing cluster start / stop instructions.

[0092] In specific implementation, please refer to Figure 3 , when the user account logging in to the client has administrative privileges (i.e., the user account logging in to the client is an administrator account), in response to the client's operation of opening the target data warehouse, the metadata management host enters the data warehouse configuration mode.

[0093] The metadata management host in the data warehouse configuration mode is used to receive data warehouse configuration instructions; the data warehouse configuration instructions include environment deployment instructions, role grouping addition instructions, and computing cluster control instructions. The metadata management host executes the data warehouse configuration instructions, triggering the target data warehouse to store the metadata associated with the preset data processing components according to the metadata storage structure. Specifically, as Figure 3 shown, after the client controls the metadata management host to start, the client can deploy the environment, add clusters, add role groupings, generate role instances, configure clusters, start or stop clusters, run / stop instances, and monitor the instance status through the metadata management host. For the convenience of those skilled in the art to understand, Figure 5 a schematic diagram of the business process abstraction model is also provided, and this business process abstraction model is a model obtained by abstracting and modeling the business processes associated with the Figure 3 business models.

[0094] The technical solution of this embodiment can simplify cross-platform data management operations, effectively reduce data duplicate storage and redundant calculations, thereby reducing data interconnection costs, releasing computing, storage, and human resources, and solving the metadata management problems existing in the current big data platform system.

[0095] In an exemplary embodiment, the method further includes: obtaining data warehouse modeling information input for the big data computing platform; constructing a target data warehouse according to the data warehouse hierarchical structure characterized by the data warehouse modeling information.

[0096] Among them, the data warehouse modeling information is used to characterize the data warehouse hierarchical structure of the data warehouse model.

[0097] In an exemplary embodiment, as Figure 6 shown, the data warehouse model includes at least one of a data access layer (ODS introduction layer), a data processing layer (CDM common layer), or a data application layer (ADS application layer).

[0098] Among them, the data access layer is used to access the original business data. For example Figure 7 as shown, the original business data includes at least one of the configuration metadata of the big data computing platform, system operation data, cluster operation data, storage metadata, computing job data, log data, or cluster operation data.

[0099] Among them, the data processing layer includes the data warehouse detail layer, the data warehouse summary layer, and the dimension common layer.

[0100] In practical applications, the method of data warehouse dimensional modeling can be adopted for dimension design and fact table design. Including but not limited to the following dimension tables, as shown in Table 6:

[0101]

[0102] Table 6

[0103] Including but not limited to the following fact tables, as shown in Table 7:

[0104]

[0105] Table 7

[0106] Combined with the ODS introduction layer and the DWS summary layer, a hierarchical model design is carried out, as shown in Tables 8 and 9:

[0107]

[0108] Table 8

[0109]

[0110] Table 9

[0111] Among them, the data warehouse detail layer is used to convert the original business data into a detail fact table according to the preset business logic; the data warehouse summary layer is used to summarize the detail data in the detail fact table according to the dimension information provided by the dimension common layer to obtain summary data.

[0112] For example Figure 7 as shown, the data application layer is used to provide at least one of the data insight function, file merging function, data middle platform function, operation and maintenance monitoring function, parameter recommendation function, or data lake warehouse management function.

[0113] In specific implementation, the metadata management host can obtain the data warehouse modeling information input for the big data computing platform, and the metadata management host can construct a target data warehouse according to the data warehouse hierarchical structure represented by the data warehouse modeling information.

[0114] The technical solution of this embodiment can effectively combine metadata with business data to provide a unified data warehouse structure for the service recipient to support different business scenarios of the service recipient.

[0115] In an exemplary embodiment, the metadata storage structure includes metadata data domains divided according to at least two preset data domain levels; the metadata data domains include at least one of a configuration domain, a cost domain, a storage domain, a computing domain, an operation domain, or a stability domain; each metadata data domain is associated with at least one sub-data domain.

[0116] In a specific implementation, the metadata storage structure can divide metadata into multiple first-level data domains - a configuration domain, a cost domain, a storage domain, a computing domain, an operation domain, and a stability domain. Each first-level data domain contains several second-level data domains, defining the content of each data domain and the information it covers in detail, comprehensively covering the metadata requirements of the big data platform.

[0117] In practical applications, the hierarchical relationship between the data domains in the metadata storage structure can be as shown in Table 10 below:

[0118]

[0119] Table 10

[0120] In another embodiment, as Figure 8 shown, a method for operating metadata of a big data computing platform is provided. Taking the case where this method is applied to the Figure 1 terminal as an example, it includes the following steps:

[0121] Step S802, in response to the client's access operation to the target data warehouse, obtain the service access verification ticket submitted by the client; the service access verification ticket is generated according to a preset network authentication protocol and is used to verify the account identity of the user account.

[0122] Step S804, perform an identity verification operation on the client according to the service access verification ticket.

[0123] Step S806, in response to the verification passed event triggered by the identity verification operation, establish a communication connection with the target computing cluster.

[0124] Step S808, execute the metadata operation instruction input by the client for the target metadata object, and determine the storage location information of the target metadata object corresponding to the metadata storage structure.

[0125] Step S810: According to the storage location information, control the target computing cluster to perform corresponding metadata operations on the target metadata object, and obtain the metadata operation result of the metadata operation instruction.

[0126] Step S812: Return the metadata operation result of the metadata operation instruction to the client, and disconnect the communication connection with the target computing cluster.

[0127] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a metadata operation method for a big data computing platform described above.

[0128] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0129] Based on the same inventive concept, an embodiment of the present application further provides a metadata operation device for a big data computing platform for implementing the metadata operation method for a big data computing platform involved above. The implementation solution provided by this device to solve the problem is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more embodiments of the metadata operation device for a big data computing platform provided below can refer to the limitations of the metadata operation method for a big data computing platform described above, and will not be repeated here.

[0130] In an exemplary embodiment, as Figure 9 shown, a metadata operation device for a big data computing platform is provided, including:

[0131] An access module 910, configured to establish a communication connection with a target computing cluster in response to an access operation of a client to a target data warehouse; the target data warehouse is used to store metadata associated with a preset data processing component according to a preset metadata storage structure; the preset data processing component includes at least two data processing components associated with a preset big data computing platform; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation;

[0132] An execution module 920 is configured to execute the metadata operation instruction input by the client for the target metadata object, determine the storage location information of the target metadata object corresponding to the metadata storage structure, and control the target computing cluster to perform a corresponding metadata operation on the target metadata object according to the storage location information, so as to obtain the metadata operation result of the metadata operation instruction.

[0133] A return module 930 is configured to return the metadata operation result of the metadata operation instruction to the client and disconnect the communication connection with the target computing cluster.

[0134] In one embodiment, when the user account logged in to the client does not have management authority, the access module 910 is specifically configured to, in response to an access operation of the client to the target data warehouse, obtain a service access verification ticket submitted by the client; the service access verification ticket is a ticket for verifying the account identity of the user account generated according to a preset network authentication protocol; perform an identity verification operation on the client according to the service access verification ticket; and establish a communication connection with the target computing cluster in response to a verification passed event triggered by the identity verification operation.

[0135] In one embodiment, when the user account logged in to the client has management authority, the device is further configured to enter a data warehouse configuration mode in response to an open operation of the client to the target data warehouse; the host in the data warehouse configuration mode is configured to receive a data warehouse configuration instruction; the data warehouse configuration instruction includes at least one of an environment deployment instruction, a role group addition instruction, a computing cluster control instruction, an instance state control instruction, and an instance state monitoring instruction; execute the data warehouse configuration instruction, and trigger the target data warehouse to store the metadata associated with the preset data processing component according to the metadata storage structure.

[0136] In one embodiment, the device is further configured to obtain data warehouse modeling information input for the big data computing platform; the data warehouse modeling information is used to represent the data warehouse hierarchical structure of the data warehouse model; and construct the target data warehouse according to the data warehouse hierarchical structure represented by the data warehouse modeling information.

[0137] In one embodiment, the data warehouse model includes at least one of a data access layer, a data processing layer, or a data application layer; the data access layer is used to access original business data; the original business data includes at least one of configuration metadata, system operation data, cluster operation data, storage metadata, computing job data, log data, or cluster operation data of the big data computing platform; the data processing layer includes a data warehouse detail layer, a data warehouse summary layer, and a dimension common layer; the data warehouse detail layer is used to convert the original business data into a detail fact table according to a preset business logic; the data warehouse summary layer is used to summarize the detail data in the detail fact table according to the dimension information provided by the dimension common layer to obtain summary data; the data application layer is used to provide at least one of a data insight function, a file merging function, a data middle platform function, an operation and maintenance monitoring function, a parameter recommendation function, or a data lake warehouse management function.

[0138] In one embodiment, the metadata storage structure includes metadata data domains divided according to at least two preset data domain levels; the metadata data domains include at least one of a configuration domain, a cost domain, a storage domain, a computing domain, an operation domain, or a stability domain; each metadata data domain is associated with at least one sub-data domain.

[0139] Each module in the above metadata operation device of the big data computing platform can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0140] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 10 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a metadata operation method of a big data computing platform.

[0141] Those skilled in the art can understand that Figure 10 the structure shown in Figure 10 is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0142] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the above-mentioned method for operating metadata of a big data computing platform. Here, the steps of the method for operating metadata of a big data computing platform may be the steps in the method for operating metadata of a big data computing platform in each of the above embodiments.

[0143] In one embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the above-mentioned method for operating metadata of a big data computing platform. Here, the steps of the method for operating metadata of a big data computing platform may be the steps in the method for operating metadata of a big data computing platform in each of the above embodiments.

[0144] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the above-mentioned method for operating metadata of a big data computing platform. Here, the steps of the method for operating metadata of a big data computing platform may be the steps in the method for operating metadata of a big data computing platform in each of the above embodiments.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant regulations.

[0146] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magneto resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0147] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in the present application.

[0148] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for operating metadata of a big data computing platform, characterized in that, The method includes: In response to an access operation of a client to a target data warehouse, establishing a communication connection with a target computing cluster; the target data warehouse is used to store metadata associated with a preset data processing component according to a preset metadata storage structure; the preset data processing component includes at least two data processing components associated with a preset big data computing platform; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation. Executing a metadata operation instruction input by the client for a target metadata object, determining storage location information corresponding to the target metadata object in the metadata storage structure, and according to the storage location information, controlling the target computing cluster to perform a corresponding metadata operation on the target metadata object to obtain a metadata operation result of the metadata operation instruction. Returning the metadata operation result of the metadata operation instruction to the client and disconnecting the communication connection with the target computing cluster.

2. The method according to claim 1, wherein In the case where the user account logging in to the client does not have management authority, the establishing a communication connection with a target computing cluster in response to an access operation of the client to the target data warehouse includes: In response to an access operation of the client to the target data warehouse, obtaining a service access verification ticket submitted by the client; the service access verification ticket is generated according to a preset network authentication protocol and is used to verify the account identity of the user account. Performing an identity verification operation on the client according to the service access verification ticket. In response to a verification passed event triggered by the identity verification operation, establishing a communication connection with the target computing cluster.

3. The method according to claim 1, characterized in that, In the case where the user account logging in to the client has management authority, the method further includes: In response to an opening operation of the client to the target data warehouse, entering a data warehouse configuration mode; the host in the data warehouse configuration mode is used to receive data warehouse configuration instructions; the data warehouse configuration instructions include at least one of an environment deployment instruction, a role grouping addition instruction, a computing cluster control instruction, an instance status control instruction, and an instance status monitoring instruction. Executing the data warehouse configuration instruction to trigger the target data warehouse to store the metadata associated with the preset data processing component according to the metadata storage structure.

4. The method according to claim 1, wherein The method further includes: Obtaining data warehouse modeling information input for the big data computing platform; the data warehouse modeling information is used to represent the data warehouse hierarchical structure of a data warehouse model. Constructing the target data warehouse according to the data warehouse hierarchical structure represented by the data warehouse modeling information.

5. The method according to claim 4, characterized in that, The data warehouse model includes at least one of a data access layer, a data processing layer, or a data application layer. The data access layer is used to access original business data; the original business data includes at least one of configuration metadata, system operation data, cluster operation data, storage metadata, computing job data, log data, or cluster operation data of the big data computing platform. The data processing layer includes a data warehouse detail layer, a data warehouse summary layer, and a dimension common layer; the data warehouse detail layer is used to convert the original business data into a detail fact table according to a preset business logic; the data warehouse summary layer is used to summarize the detail data in the detail fact table according to the dimension information provided by the dimension common layer to obtain summary data; The data application layer is used to provide at least one of a data insight function, a file merging function, a data middle platform function, an operation and maintenance monitoring function, a parameter recommendation function, or a data lake warehouse management function.

6. The method according to claim 1, characterized in that The metadata storage structure includes metadata data domains divided according to at least two preset data domain levels; the metadata data domains include at least one of a configuration domain, a cost domain, a storage domain, a computing domain, an operation domain, or a stability domain; each of the metadata data domains is associated with at least one sub-data domain.

7. A metadata operation device for a big data computing platform, characterized in that, The device includes: An access module, configured to establish a communication connection with a target computing cluster in response to an access operation of a client to a target data warehouse; the target data warehouse is used to store metadata associated with a preset data processing component according to a preset metadata storage structure; the preset data processing component includes at least two data processing components associated with a preset big data computing platform; the target computing cluster is the computing cluster corresponding to the data processing component pointed to by the access operation; An execution module, configured to execute a metadata operation instruction input by the client to a target metadata object, determine the storage location information corresponding to the target metadata object in the metadata storage structure, and according to the storage location information, control the target computing cluster to perform a corresponding metadata operation on the target metadata object to obtain a metadata operation result of the metadata operation instruction; A return module, configured to return the metadata operation result of the metadata operation instruction to the client and disconnect the communication connection with the target computing cluster.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images