Metadata model and implementation method based on OLAP query platform with billions of data

Abstract

The present invention discloses a metadata model based on a 10 billion data OLAP query platform and an implementation method thereof, the method comprising: presetting a data storage layer, a data mapping layer, a data processing layer, and a data display layer connected in sequence; storing each business data in different storage media of the data storage layer; performing relationship mapping on various business data stored in the data storage layer to form a program object; based on the relationship mapping of the data mapping layer, reflecting various business data stored in the data storage layer through objects, extracting class attributes and methods in the objects, filtering and finally storing the attributes of the objects in a Mysql database; outputting and displaying the object attributes of the Mysql database, and providing the data model for external calls. The present invention implements a set of metadata models, and users only need to query data through metadata services without knowing the underlying data storage, thereby improving the data storage processing efficiency and providing convenience for users.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and processing, and in particular to a metadata model based on an OLAP query platform with tens of billions of data and an implementation method thereof. Background Art

[0002] With the development of science and technology and the continuous improvement of informatization level, many companies now use information processing platforms, such as the OLAPP platform.

[0003] OLAP generally refers to online analytical processing. Many companies have OLAP platforms, but most OLAPs are implemented based on one or more storage media, and are directly connected to queries without an intermediate conversion layer. As a result, different departments and businesses need to specify storage media when extracting data, and specify database tables for query. It is impossible to map through the metadata model, resulting in low data query processing efficiency and inconvenience for users.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a metadata model based on an OLAP query platform for billions of data and its implementation method, server and storage medium are provided. The present invention implements a set of metadata models, and users only need to query data through metadata services without knowing the underlying data storage, thereby improving the efficiency of data storage processing and providing convenience for users.

[0006] In order to solve the above technical problems, the first aspect of the present invention provides a method for implementing a metadata model based on a 10 billion data OLAP query platform, the method comprising:

[0007] A metadata model based on a 10 billion data OLAP query platform, including:

[0008] The data storage layer, data mapping layer, data processing layer, and data display layer are set up in sequence;

[0009] The data storage layer includes a plurality of different storage media, and the data storage layer is used to store various business data;

[0010] The data mapping layer is connected to the data storage layer, and the data mapping layer is used to perform relationship mapping on various business data stored in the data storage layer to form a program object, so as to generate implementation classes of different storage media through metadata templates;

[0011] The data processing layer is connected to the data mapping layer. The data processing layer is used to reflect various business data stored in the data storage layer through objects based on the relationship mapping of the data mapping layer, extract class attributes and methods in the objects, filter out unnecessary methods, and finally store the attributes of the objects in the Mysql database; so as to obtain the attributes and methods in the class through the reflection mechanism;

[0012] The data display layer is used to output and display the object attributes of the Mysql database, and to make the data model of the data table available for external calls to provide an operation interface for business personnel.

[0013] The metadata model based on the 10 billion data OLAP query platform, wherein the data storage layer includes: Hive data table, Doris data table, Mysql data table;

[0014] The Mysql data table is used to store report data of millions of data sets, the Doris data table is used to store model data of hundreds of millions, and the Hive data table is used to store log data of tens of billions.

[0015] The metadata model based on the 10 billion data OLAP query platform is characterized in that the data mapping layer includes:

[0016] A metadata template, and a Hive table implementation class, a Doris table implementation class, and a Mysql table implementation class respectively connected to the metadata template;

[0017] The metadata template is provided with an attribute acquisition method, a library address attribute, and a table name attribute, which are used to generate implementation classes of different storage media through the metadata template;

[0018] The Hive table implementation class, the Doris table implementation class, and the Mysql table implementation class respectively inherit the metadata template to obtain the attribute acquisition method, the library address attribute, and the table name attribute, and at the same time add the field attribute and the storage attribute method.

[0019] The metadata model based on the 10 billion data OLAP query platform, wherein the data processing layer includes: a class reflection processing unit, a class attribute and method filtering unit, and a polymorphic attribute value unit connected in sequence;

[0020] The class reflection processing unit is used to obtain methods and properties in the implementation class through the reflection mechanism of the class;

[0021] The class attribute and method filtering unit is used to obtain the corresponding relationship used within the platform through attribute and method filtering.

[0022] The polymorphic attribute value acquisition unit is used to obtain the externally exposed fields through a polymorphic retrieval method, and the corresponding relationship is used externally by the platform.

[0023] The metadata model based on the 10 billion data OLAP query platform, wherein the data display layer includes: a metadata external model, and a Hive model, a Doris model, and a Mysql model respectively connected to the metadata external model.

[0024] A method for implementing a metadata model based on a 10 billion data OLAP query platform as described in any of the above items, wherein the implementation method comprises:

[0025] Pre-set a data storage layer, a data mapping layer, a data processing layer, and a data display layer that are connected in sequence; set the data storage layer to include a variety of different storage media, and the data storage layer is used to store various business data;

[0026] Store each business data into different storage media in the data storage layer according to the business needs of each department;

[0027] Map the various business data stored in the data storage layer to form a program object, so as to generate implementation classes of different storage media through metadata templates;

[0028] Based on the relational mapping of the data mapping layer, various business data stored in the data storage layer are reflected through objects, class attributes and methods in the objects are extracted, and unnecessary methods are filtered out, and finally the attributes of the objects are stored in the MySQL database;

[0029] The object attributes of the Mysql database are output and displayed, and the data model of the data table is available for external calls to provide an operation interface for business personnel.

[0030] The method for implementing the metadata model based on the 10 billion data OLAP query platform, wherein the step of storing each business data in different storage media of the data storage layer according to the business needs of each department includes:

[0031] The report data of millions of data sets are stored in the MySQL data table of the data storage layer, the model data of hundreds of millions are stored in the Doris data table of the data storage layer, and the log data of tens of billions are stored in the Hive data table of the data storage layer.

[0032] The method for implementing the metadata model based on the 10 billion data OLAP query platform, wherein the step of mapping the various business data stored in the data storage layer to form a program object and generating implementation classes of different storage media through the metadata template includes:

[0033] The metadata template of the data mapping layer is set to obtain the attribute method, the library address attribute, and the table name attribute;

[0034] Make the table implementation class of the data mapping layer inherit the metadata template's acquisition attribute method, library address attribute, and table name attribute, and add field attributes and storage attribute methods;

[0035] Set the call storage property method in the task startup class;

[0036] In the storage property method, all properties and methods in this class are obtained through the reflection mechanism, and the remaining useless methods are filtered out;

[0037] Store the attributes in the implementation class to form metadata in the Mysql table.

[0038] A server includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including the method for executing any one of the methods described above.

[0039] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a server, the server is enabled to perform any one of the methods described above.

[0040] Beneficial effects: Compared with the prior art, the present invention provides a method for implementing a metadata model based on an OLAP query platform with billions of data. The present invention implements the metadata model of the OLAP query platform through reflection and object-oriented polymorphism. The same set of metadata models is used to implement table data of multiple storage media. The present invention can achieve unified specifications for table structures of different storage media through mapping through the metadata model; allowing users to use a unified query method, so that users do not need to know which storage medium and which table the OLAP underlying data is in, to achieve unified specifications and unified entry, so that users only need to query data through metadata services, and do not need to know the underlying data storage, which improves the efficiency of data storage processing and provides convenience for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the functional principle frame structure of the metadata model based on the OLAP query platform for tens of billions of data provided by an embodiment of the present invention.

[0042] Figure 2 A schematic diagram of the layer structure of the metadata model based on the 10 billion data OLAP query platform provided in an embodiment of the present invention.

[0043] Figure 3A flowchart of the implementation of the data mapping layer and the data storage layer based on the metadata model implementation method of the 10 billion data OLAP query platform provided in an embodiment of the present invention.

[0044] Figure 4 It is a first structural diagram of the visualization operation of the data display layer based on the metadata model of the 10 billion data OLAP query platform according to an embodiment of the present invention.

[0045] Figure 5 It is a second structural diagram of the visualization operation of the data display layer based on the metadata model of the 10 billion data OLAP query platform according to an embodiment of the present invention.

[0046] Figure 6 It is a flowchart of a method for implementing a metadata model based on a 10 billion data OLAP query platform according to an embodiment of the present invention.

[0047] Figure 7 It is a block diagram of the internal structure principle of the server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] With the development of science and technology and the continuous improvement of informatization level, many companies now use information processing platforms, such as the OLAPP platform.

[0050] OLAP generally refers to online analytical processing. Many companies have OLAP platforms, but most OLAPs are implemented based on one or more storage media, and are directly connected to queries without an intermediate conversion layer. As a result, different departments and businesses need to specify storage media when extracting data, and specify database tables for query. It is impossible to map through the metadata model, resulting in low data query processing efficiency and inconvenience for users.

[0051] In order to solve the problems of the prior art, this embodiment provides a method for implementing a metadata model based on a 10 billion data OLAP query platform. When implemented, the metadata model is divided into four layers, namely, a data storage layer, a data mapping layer, a data processing layer, and a data display layer. The data storage layer refers to different storage media; the data mapping layer refers to the implementation class of different storage media generated by metadata templates; the data processing layer refers to obtaining the attributes and methods in the class through the reflection mechanism; the data display layer refers to the data model of the data table for external calls. The present invention implements the metadata model of the OLAP query platform through reflection and object-oriented polymorphism. The same set of metadata models is used to implement the table data of multiple storage media. The present invention can use the metadata model to map the table structure of different storage media to achieve unified specifications; let users use a unified query method, so that users do not need to know which storage medium and which table the OLAP underlying data is in, to achieve unified specifications and unified entry, so that users only need to query data through metadata services, and do not need to know the underlying data storage, which improves the efficiency of data storage processing and provides convenience for users.

[0052] Exemplary Devices

[0053] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a metadata model based on a 10 billion data OLAP query platform, including:

[0054] The data storage layer, data mapping layer, data processing layer, and data display layer are set up in sequence;

[0055] The data storage layer includes a variety of different storage media, and the data storage layer is used to store various business data; specifically, Figure 1 As shown, the data storage layer includes: Hive data table, Doris data table, and MySQL data table; wherein the MySQL data table is used to store report data of millions of data sets, the Doris data table is used to store model data at the level of hundreds of millions, and the Hive data table is used to store log data at the level of tens of billions.

[0056] In the embodiment of the present invention, the data mapping layer is connected to the data storage layer, and the data mapping layer is used to perform relationship mapping on various business data stored in the data storage layer to form a program object, so as to generate implementation classes of different storage media through metadata templates.

[0057] The data processing layer is connected to the data mapping layer. The data processing layer is used to reflect various business data stored in the data storage layer through objects based on the relationship mapping of the data mapping layer, extract class attributes and methods in the objects, filter out unnecessary methods, and finally store the attributes of the objects in the Mysql database; so as to obtain the attributes and methods in the class through the reflection mechanism.

[0058] The data display layer is used to output and display the object attributes of the Mysql database and provide the data model of the data table for external calls to provide an operation interface for business personnel. Figure 1 As shown, the data presentation layer includes: a metadata external model, and a Hive model, a Doris model, and a Mysql model respectively connected to the metadata external model.

[0059] That is, in the embodiment of the present invention, the metadata model is divided into 4 layers, namely, the data storage layer, the data mapping layer, the data processing layer, and the data display layer. The data storage layer refers to different storage media, and the data storage layer stores the company's business data, such as indicator data, dashboard data, model data, etc. The data mapping layer refers to the implementation class that generates different storage media through metadata templates. The data mapping layer performs relationship mapping on various information in the storage to form a program object. The data processing layer refers to obtaining the attributes and methods in the class through the reflection mechanism. The data processing layer of the present invention extracts the attributes and methods in the object by reflecting the object, and filters the unnecessary methods at the same time, and finally stores the attributes of the object in the Mysql database. The data display layer refers to the data model of the data table for external calls. The data display layer of the present invention can display the object attributes of the Mysql database, which is convenient for business personnel to operate.

[0060] As can be seen from the above, the present invention implements a set of metadata models. Users only need to query data through metadata services and do not need to know the underlying data storage, which improves the data storage processing efficiency and provides convenience for users.

[0061] Furthermore, if Figure 1 As shown, the data storage layer in the embodiment of the present invention has different storage media (such as Hive, Doris, Mysql), each of which has its own query and storage advantages and disadvantages, and also has differences in the diversity of field types. Therefore, the business data is stored in different media according to the business needs of each department, so that the business can extract data faster.

[0062] Among them, Hive is a data warehouse tool, Doris is an analytical data product, and Mysql is a relational database. The present invention combines the advantages and disadvantages of the performance of various storage media. Mysql stores report data of millions of data sets, Doris stores model data of hundreds of millions, and Hive stores log data of tens of billions. Storage is performed according to the company's business requirements.

[0063] Furthermore, if Figure 1 and Figure 3 As shown, the data mapping layer includes: a metadata template, and a Hive table implementation class, a Doris table implementation class, and a Mysql table implementation class connected to the metadata template respectively; the metadata template is provided with an acquisition attribute method, a library address attribute, and a table name attribute, which is used to generate an implementation class of different storage media through the metadata template; the Hive table implementation class, the Doris table implementation class, and the Mysql table implementation class respectively inherit the metadata template to obtain the acquisition attribute method, the library address attribute, and the table name attribute, and add multiple field attributes (such as field attribute 01, field attribute 02, and field attribute 03) and storage attribute methods; and the Hive table implementation class, the Doris table implementation class, and the Mysql table implementation class are respectively connected to the task startup class, and the task startup class includes the Hive table implementation class object startup storage method and the Doris table implementation class object startup storage method. The data mapping layer in the embodiment of the present invention: the fields of the table are mapped to the implementation class, and the data foundation needs to be prepared before the metadata model. The comparison before mapping (Table 1) and after mapping (Table 2) is as follows:

[0064] Table 1 Basic information of data sheet:

[0065]

[0066] Table 2 Basic information of implementation class

[0067]

[0068] Furthermore, if Figure 1 As shown, the data processing layer includes: a class reflection processing unit, a class attribute and method filtering unit, and a polymorphic attribute value unit connected in sequence; the class reflection processing unit is used to obtain methods and attributes in the implementation class through the reflection mechanism of the class; the class attribute and method filtering unit is used to obtain the corresponding relationship used internally by the platform through attribute and method filtering; the polymorphic attribute value unit is used to obtain the externally exposed fields through the polymorphic extraction method, and the corresponding relationship used externally by the platform. The data processing layer of the embodiment of the present invention: obtains the methods and attributes in the implementation class through the reflection mechanism of the class, and obtains the corresponding relationship used internally by the platform through attribute and method filtering (as shown in Table 3 below). The externally exposed fields are obtained through the polymorphic extraction method, and the corresponding relationship is used externally by the platform (as shown in Table 4 below).

[0069] Table 3 Correspondence between internal usage of the platform:

[0070]

[0071] Table 4 Platform external usage correspondence:

[0072]

[0073]

[0074]

[0075] Furthermore, if Figure 1 As shown, the data display layer includes: a metadata external model, and a Hive model, a Doris model, and a Mysql model respectively connected to the metadata external model. The data display layer of the embodiment of the present invention: through the data of the corresponding relationship used by the platform externally, operations can be performed through visual operations, such as Figure 4 and Figure 5 shown.

[0076] As can be seen from the above, the present invention realizes the OLAP query platform metadata model through reflection and object-oriented polymorphism, and uses the same set of metadata models to realize table data of multiple storage media.

[0077] Exemplary Methods

[0078] Based on the above metadata model embodiment, the embodiment of the present invention also provides a method for implementing the metadata model based on the OLAP query platform of billions of data as described in any of the above items. The method of this embodiment can be applied to the server. When it is implemented specifically, Figure 6 As shown in the embodiment of the present invention, the metadata model implementation method based on the 10 billion data OLAP query platform specifically includes the following steps:

[0079] Step S100, pre-setting a data storage layer, a data mapping layer, a data processing layer, and a data display layer connected in sequence; setting the data storage layer to include a variety of different storage media, and the data storage layer is used to store various business data;

[0080] Step S200: storing each business data into different storage media in the data storage layer according to the business needs of each department;

[0081] Step S300: various business data stored in the data storage layer are mapped to form a program object, so as to generate implementation classes of different storage media through metadata templates;

[0082] Step S400: Based on the relationship mapping of the data mapping layer, various business data stored in the data storage layer are reflected through objects, class attributes and methods in the objects are extracted, unnecessary methods are filtered, and finally the attributes of the objects are stored in the MySQL database;

[0083] Step S500: Output and display the object attributes of the Mysql database, and make the data model of the data table available for external calls to provide an operation interface for business personnel, as described above.

[0084] The step of storing each business data in different storage media of the data storage layer according to the business needs of each department includes:

[0085] The report data of millions of data sets are stored in the MySQL data table of the data storage layer, the model data of hundreds of millions are stored in the Doris data table of the data storage layer, and the log data of tens of billions are stored in the Hive data table of the data storage layer.

[0086] The step of mapping the various business data stored in the data storage layer to form a program object and generating implementation classes of different storage media through metadata templates includes:

[0087] The metadata template of the data mapping layer is set to obtain the attribute method, the library address attribute, and the table name attribute;

[0088] Make the table implementation class of the data mapping layer inherit the metadata template's acquisition attribute method, library address attribute, and table name attribute, and add field attributes and storage attribute methods;

[0089] Set the call storage property method in the task startup class;

[0090] In the storage property method, all properties and methods in this class are obtained through the reflection mechanism, and the remaining useless methods are filtered out;

[0091] Store the attributes in the implementation class to form metadata in the Mysql table.

[0092] As can be seen from the above, in the embodiment of the present invention, the metadata model is pre-divided into four layers, namely, the data storage layer, the data mapping layer, the data processing layer, and the data display layer; wherein the data storage layer includes different storage media; the data mapping layer refers to the implementation class for generating different storage media through the metadata template; the data processing layer refers to obtaining the attributes and methods in the class through the reflection mechanism; the data display layer refers to the data model of the data table for external calls.

[0093] The logical relationship between the various layers of the metadata model in the embodiment of the present invention is as follows: Figure 2As shown, from the bottom layer to the surface layer, they are the data storage layer, data mapping layer, data processing layer, and data display layer:

[0094] The data storage layer is used to store business data, such as indicator data, dashboard data, model data, etc.

[0095] The data mapping layer is used to map the relationships between various information in the storage to form a program object;

[0096] The data processing layer extracts the properties and methods of the object by reflecting the object, and filters out the unnecessary methods, and finally stores the properties of the object in the MySQL database;

[0097] The data display layer displays the object properties of the MySQL database to facilitate operations by business personnel.

[0098] Furthermore, if Figure 2 and 3 As shown, in the embodiment of the present invention, the data mapping layer and the data storage layer implement the following steps:

[0099] Step S21, construct a metadata template, and set the attribute acquisition method, library address attribute, and table name attribute in the metadata template;

[0100] Step S22: Inherit the metadata template of each table implementation class to obtain the original properties and methods, and add field properties and storage property methods;

[0101] Step S23, calling the storage attribute method in the task startup class;

[0102] Step S24: in the storage attribute method, all attributes and methods in the class are obtained through the reflection mechanism, and the remaining useless methods are filtered out;

[0103] Step S25: Store the attributes in the implementation class to form metadata in the Mysql table.

[0104] Based on the above embodiment, the present invention further provides a server, whose principle block diagram can be shown as follows: Figure 7As shown. The server includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the server is used to provide computing and control capabilities. The memory of the server includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the server is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a metadata model implementation method based on a 10 billion data OLAP query platform is implemented. The display screen of the server can be a liquid crystal display or an electronic ink display.

[0105] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the server to which the solution of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0106] In one embodiment, a server is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:

[0107] Pre-set a data storage layer, a data mapping layer, a data processing layer, and a data display layer that are connected in sequence; set the data storage layer to include a variety of different storage media, and the data storage layer is used to store various business data;

[0108] Store each business data into different storage media in the data storage layer according to the business needs of each department;

[0109] Map the various business data stored in the data storage layer to form a program object, so as to generate implementation classes of different storage media through metadata templates;

[0110] Based on the relational mapping of the data mapping layer, various business data stored in the data storage layer are reflected through objects, class attributes and methods in the objects are extracted, and unnecessary methods are filtered out, and finally the attributes of the objects are stored in the MySQL database;

[0111] The object attributes of the Mysql database are output and displayed, and the data model of the data table is available for external calls to provide an operation interface for business personnel.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] In summary, the present invention discloses a metadata model based on a 10 billion data OLAP query platform and its implementation method, server and storage medium, pre-setting a data storage layer, a data mapping layer, a data processing layer and a data display layer connected in sequence; storing each business data in different storage media of the data storage layer; performing relationship mapping on various business data stored in the data storage layer to form a program object; based on the relationship mapping of the data mapping layer, reflecting various business data stored in the data storage layer through objects, extracting class attributes and methods in the objects, filtering and finally storing the attributes of the objects in a Mysql database; outputting and displaying the object attributes of the Mysql database, and providing the data model for external calls. The present invention implements a set of metadata models, and users only need to query data through metadata services without knowing the underlying data storage, thereby improving the efficiency of data storage processing and providing convenience for users.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A metadata model based on OLAP query platform with billions of data, characterized by: include: The data storage layer, data mapping layer, data processing layer, and data display layer are set up in sequence; The data storage layer includes a plurality of different storage media, and the data storage layer is used to store various business data; The data mapping layer is connected to the data storage layer, and the data mapping layer is used to perform relationship mapping on various business data stored in the data storage layer to form a program object, so as to generate implementation classes of different storage media through metadata templates; The metadata template is provided with an attribute acquisition method, a library address attribute, and a table name attribute, which are used to generate implementation classes of different storage media through the metadata template; The data processing layer is connected to the data mapping layer. The data processing layer is used to reflect various business data stored in the data storage layer through objects based on the relationship mapping of the data mapping layer, extract class attributes and methods in the objects, filter out unnecessary methods, and finally store the attributes of the objects in the MySQL database; To achieve the acquisition of class attributes and methods through the reflection mechanism; The data processing layer includes: a class reflection processing unit, a class attribute and method filtering unit, and a polymorphic attribute value obtaining unit connected in sequence; The class reflection processing unit is used to obtain methods and properties in the implementation class through the reflection mechanism of the class; The class attribute and method filtering unit is used to obtain the corresponding relationship used within the platform through attribute and method filtering; The polymorphic attribute value acquisition unit is used to obtain the externally exposed fields through the polymorphic retrieval method, and the corresponding relationship is used externally by the platform; The data processing layer obtains the methods and properties in the implementation class through the reflection mechanism of the class, and obtains the corresponding relationship used within the platform through property and method filtering; The data display layer is used to output and display the object attributes of the Mysql database, and to make the data model of the data table available for external calls to provide an operation interface for business personnel; The data display layer includes: a metadata external model, and a Hive model, a Doris model, and a Mysql model respectively connected to the metadata external model; The data display layer uses the corresponding data outside the platform and can be operated through visual operations.

2. According to claim 1, the metadata model based on the OLAP query platform of tens of billions of data is characterized in that: The data storage layer includes: Hive data table, Doris data table, and Mysql data table; The Mysql data table is used to store report data of millions of data sets, the Doris data table is used to store model data of hundreds of millions, and the Hive data table is used to store log data of tens of billions.

3. According to claim 1, the metadata model based on the OLAP query platform of tens of billions of data is characterized in that: The data mapping layer includes: A metadata template, and a Hive table implementation class, a Doris table implementation class, and a Mysql table implementation class respectively connected to the metadata template; The metadata template is provided with an attribute acquisition method, a library address attribute, and a table name attribute, which are used to generate implementation classes of different storage media through the metadata template; The Hive table implementation class, the Doris table implementation class, and the Mysql table implementation class respectively inherit the metadata template to obtain the attribute acquisition method, the library address attribute, and the table name attribute, and at the same time add the field attribute and the storage attribute method.

4. A method for implementing a metadata model based on a 10 billion data OLAP query platform as claimed in any one of claims 1 to 3, characterized in that: The implementation method includes: Pre-set a data storage layer, a data mapping layer, a data processing layer, and a data display layer that are connected in sequence; set the data storage layer to include a variety of different storage media, and the data storage layer is used to store various business data; Store each business data into different storage media in the data storage layer according to the business needs of each department; Map the various business data stored in the data storage layer to form a program object, so as to generate implementation classes of different storage media through metadata templates; Based on the relational mapping of the data mapping layer, various business data stored in the data storage layer are reflected through objects, class attributes and methods in the objects are extracted, and unnecessary methods are filtered out, and finally the attributes of the objects are stored in the MySQL database; The object attributes of the Mysql database are output and displayed, and the data model of the data table is available for external calls to provide an operation interface for business personnel.

5. The method for implementing the metadata model based on the OLAP query platform of tens of billions of data according to claim 4 is characterized in that: The step of storing each business data in different storage media of the data storage layer according to the business needs of each department includes: The report data of millions of data sets are stored in the MySQL data table of the data storage layer, the model data of hundreds of millions are stored in the Doris data table of the data storage layer, and the log data of tens of billions are stored in the Hive data table of the data storage layer.

6. The method for implementing a metadata model based on a 10 billion data OLAP query platform according to claim 4, characterized in that: The step of mapping the various business data stored in the data storage layer to form a program object and generating implementation classes of different storage media through metadata templates includes: The metadata template of the data mapping layer is set to obtain the attribute method, the library address attribute, and the table name attribute; Make the table implementation class of the data mapping layer inherit the metadata template's acquisition attribute method, library address attribute, and table name attribute, and add field attributes and storage attribute methods; Set the call storage property method in the task startup class; In the storage property method, all properties and methods in this class are obtained through the reflection mechanism, and the remaining useless methods are filtered out; Store the attributes in the implementation class to form metadata in the Mysql table.

7. A server, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 4 to 6.

8. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of the server, the server is enabled to execute the method as described in any one of claims 4 to 6.

Images