Data fusion method and device, data processing method and device based on metadata

Abstract

The present application discloses a data fusion method and device, a data processing method and device based on meta-information, a data processing method, and a computer storage medium and an electronic device. The data fusion method includes: obtaining meta-information used to describe data attributes in a data source; generating a meta-information table structure based on the meta-information of the data in the data source; generating a data table fused with the meta-information based on the meta-information table structure and a data table used to describe the data content in the data source; and then making the generated data table fused with the meta-information include the relevant content of the meta-information, so that when analyzing the data in the data source, the meta-information of the data can also be queried, analyzed, and other related operations.

Description

Technical Field

[0001] The present application relates to the field of computer applications, and more particularly to a data fusion method and device, and a data processing method and device based on metadata. The present application also relates to a data processing method; and to a computer storage medium and electronic device. Background Art

[0002] In a traditional relational database, a user defines a table and then continuously reads and writes data in the table. However, the metadata related to the table itself cannot be queried, and cannot be merged with the data in the table for query, because most of the time there is no such business requirement.

[0003] With the continuous development of the Internet, big data has become an indispensable part of life. In the data lake scenario, massive amounts of metadata are also an important source of big data, and users also have the need to query metadata in different data sources. In this context, users will have the need to correlate and analyze various data, and at the same time, they will also have related analysis needs for the metadata data corresponding to these data. Users hope to be able to output certain metadata of the database table as columns to the final result set. Before metadata fusion business data analysis was supported, the only way was to write all this metadata into the columns of the table and then analyze it, but this method has the following problems:

[0004] 1) The metadata written into the table constitutes a large amount of redundant information, which in turn causes the storage space to become very large, and most of this large amount of redundant information is repeated;

[0005] 2) This information is difficult to change. If you want to change it, you need to change all records, etc.

[0006] Based on the above content, how to perform data analysis on the meta information and how to combine the meta information with the source data for data analysis become technical problems that those skilled in the art need to solve. Summary of the invention

[0007] The present application provides a data fusion method to solve the problem in the prior art that it is impossible to combine meta information with source data for analysis.

[0008] The present application provides a data fusion method, comprising:

[0009] Get the metadata used to describe data attributes in the data source;

[0010] Generate a meta information table structure according to the meta information of the data in the data source;

[0011] A data table incorporating the meta information is generated according to the meta information table structure and a data table used to describe the data content in the data source.

[0012] In some embodiments, generating a meta information table structure according to the meta information of the data in the data source includes:

[0013] Setting column information of the meta information table structure according to the meta information of the data in the data source;

[0014] The meta information table structure is generated according to the column information.

[0015] In some embodiments, generating a data table incorporating the meta information according to the meta information table structure and a data table for describing data content in the data source includes:

[0016] A data table fused with the meta information is generated according to the column information of the meta information table structure and the data table used to describe the data content in the data source.

[0017] In some embodiments, it also includes:

[0018] Adding a meta information identifier to the column information of the set meta information table structure;

[0019] The step of generating a data table incorporating the meta information according to the meta information table structure and a data table for describing the data content in the data source comprises:

[0020] Determining the data source corresponding to the meta-information according to the meta-information identifier in the column information;

[0021] A data table incorporating the meta information is generated according to the column information having the meta information identifier and the data table of the determined data content in the data source.

[0022] In some embodiments, generating a data table incorporating the meta information according to the meta information table structure and a data table for describing data content in the data source includes:

[0023] The meta information table structure is spliced ​​into a data table used to describe the data content in the data source to generate a data table integrated with the meta information.

[0024] In some embodiments, the step of splicing the meta information table structure into a data table used to describe the data content in the data source to generate a data table integrated with the meta information includes:

[0025] The meta information table structure is spliced ​​into a data table used to describe the data content in the data source to generate a data table integrated with virtual meta information.

[0026] In some embodiments, obtaining meta information for describing data attributes in a data source includes:

[0027] According to the document information provided by the data source, obtain the meta information used to describe the data attributes in the data source.

[0028] In some embodiments, the meta information includes at least one of the following information:

[0029] Data name information;

[0030] Data creation time information;

[0031] Data modification time information;

[0032] Database table name information;

[0033] Data instance name information;

[0034] Data editor information;

[0035] Data creator information;

[0036] Data size information;

[0037] Data path information.

[0038] In some embodiments, it also includes:

[0039] Get a data table for describing the data content in the data source.

[0040] The present application also provides a data fusion device, comprising:

[0041] An acquisition unit, used for acquiring meta information used to describe data attributes in a data source;

[0042] A generating unit, used for generating a meta information table structure according to the meta information of the data in the data source;

[0043] A fusion unit is used to generate a data table fused with the meta information according to the meta information table structure and a data table used to describe the data content in the data source.

[0044] The present application also provides a data processing method based on metadata, comprising:

[0045] Get an operation request for a meta-information operation used to describe data attributes;

[0046] According to the operation request, extracting data information that meets the operation condition in the operation request from the data table integrated with the meta information;

[0047] Send data information that meets the operating conditions.

[0048] In some embodiments, it also includes:

[0049] Get the metadata used to describe data attributes in the data source;

[0050] Generate a meta information table structure according to the meta information of the data in the data source;

[0051] A data table incorporating the meta information is generated according to the meta information table structure and a data table used to describe the data content in the data source.

[0052] In some embodiments, generating a meta information table structure according to the meta information of the data in the data source includes:

[0053] Setting column information of the meta information table structure according to the meta information of the data in the data source;

[0054] The meta information table structure is generated according to the column information.

[0055] In some embodiments, generating a data table incorporating the meta information according to the meta information table structure and a data table for describing data content in the data source includes:

[0056] According to the column information of the meta information table structure and the data table used to describe the data content in the data source, a data table merging the column information is generated.

[0057] In some embodiments, it also includes:

[0058] According to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, determining the meta information that satisfies the operation condition in the operation request;

[0059] Generate a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and a data table used to describe the data content in the data source;

[0060] The extracting, according to the operation request, data information satisfying the operation condition in the operation request from the data table integrated with the meta information, comprises:

[0061] In the generated data table incorporating the meta information, data information satisfying the operation condition in the operation request is extracted.

[0062] In some embodiments, it also includes:

[0063] Setting column information of the meta information table structure according to the meta information of the data in the data source;

[0064] Generate the meta information table structure according to the column information;

[0065] The step of determining, according to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, meta information that satisfies the operation condition in the operation request includes:

[0066] In setting the column information of the meta information table structure according to the meta information of the data in the data source, determining the column information of the meta information table structure that satisfies the operation condition in the operation request;

[0067] The step of generating a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and the data table used to describe the data content in the data source comprises:

[0068] The column information of the meta information table structure determined to meet the operation condition in the operation request and the data table used to describe the data content in the data source are combined to generate a data table fused with the meta information.

[0069] In some embodiments, it also includes:

[0070] Adding a meta information identifier to the column information of the set meta information table structure;

[0071] The step of generating a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and the data table used to describe the data content in the data source comprises:

[0072] Determining the data source according to the meta information identifier in the column information;

[0073] A data table incorporating the meta information is generated according to the column information having the meta information identifier and the data table of the determined data content in the data source.

[0074] In some embodiments, including:

[0075] An acquisition unit, used for acquiring an operation request including meta information for describing data attributes;

[0076] An extraction unit, configured to extract data information satisfying an operation condition in the operation request from a data table integrated with the meta-information according to the operation request;

[0077] The sending unit is used to send data information that meets the operating conditions.

[0078] The present application also provides a data processing method, comprising:

[0079] The computing node obtains an operation request from the front-end node for meta-information used to describe data attributes;

[0080] The computing node extracts data information that meets the operation conditions in the operation request from the data table integrated with the meta-information for the operation request for describing the data attributes;

[0081] The computing node sends the extracted data information satisfying the operation condition in the operation request to the front-end node.

[0082] The present application also provides a computer storage medium for storing a program;

[0083] When the program is read and executed by the processor, the processor executes the steps in the data fusion method described above, or the steps in the data processing method based on metadata described above, or the steps in the data processing method described above.

[0084] The present application also provides an electronic device, including:

[0085] processor;

[0086] The memory is used to store a program, which, when read and executed by the processor, enables the electronic device to execute the steps in the data fusion method as described above, or the steps in the data processing method based on metadata as described above, or the steps in the data processing method as described above.

[0087] Compared with the prior art, this application has the following advantages:

[0088] The present application provides a data fusion method, which obtains meta-information used to describe data attributes in a data source, generates a meta-information table structure based on the meta-information, and then generates a data table fused with the meta-information based on the meta-information table structure and a data table used to describe the data content in the data source; thereby, the data table includes relevant content of the meta-information, so that when analyzing the data in the data source, the meta-information of the data can also be queried, analyzed, and other related operations.

[0089] The present application also provides a data processing method based on metadata, which obtains an operation request for the metadata operation used to describe data attributes, and according to the operation request, extracts data information that meets the operation conditions in the operation request from a data table integrated with the metadata, and sends the data information that meets the operation conditions; then, it is possible to find the data information that meets the operation conditions in the data table integrated with the metadata and send it, thereby completing the real-time supply of data flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 is a flow chart of an embodiment of a data fusion method provided by the present application;

[0091] Figure 2It is a schematic diagram of setting a meta information table structure for different source data in an embodiment of a data fusion method provided by the present application;

[0092] Figure 3 It is a structural schematic diagram of an embodiment of a data fusion device provided by the present application;

[0093] Figure 4 is a flow chart of an embodiment of a data processing method based on meta-information provided by the present application;

[0094] Figure 5 It is a structural schematic diagram of an embodiment of a data processing device based on meta-information provided by the present application;

[0095] Figure 6 is a flow chart of an embodiment of a data processing method provided by the present application;

[0096] Figure 7 It is a structural diagram of a data lake computing cluster of a data processing method provided in this application. DETAILED DESCRIPTION

[0097] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0098] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0099] Based on the problem in the prior art that the meta information of the data content stored in the database cannot be obtained, this application proposes a method for fusing the meta information with the data content of the source data, so as to obtain the meta information and the corresponding data content. The details are as follows:

[0100] Please refer to Figure 1 As shown, Figure 1 is a flow chart of an embodiment of a data fusion method provided by the present application, wherein the fusion method comprises:

[0101] Step S101: Obtaining meta-information for describing data attributes in a data source.

[0102] The meta information in step S101 refers to information used to describe data attributes, such as data creation time, data modification time, data creation time, data path information, etc. Different data sources can provide different meta information. In other words, different data sources provide different data and meta information. Therefore, obtaining the meta information used to describe data attributes in the data source in step S101 includes:

[0103] According to the document information provided by the data source, obtain the meta-information in the data source that is used to describe the data attributes. Different data sources can provide different meta-information. For example, when the data source type is an object storage type, the meta-information that can be provided includes: data name (also called file name), data creation time, data last modification time and other information; when the data source type is a table storage type, the meta-information that can be provided includes: data table name, data instance name, data table creation time and other information. It can be further known that the meta-information is a kind of information that describes the data attributes in the data source. This information does not exist in the data table in the data source. Only data content is stored in the data table. The above meta-information is only an example for illustration. Any information that describes the data attributes in the data source can be regarded as meta-information. Therefore, the meta-information may include at least one of the following information:

[0104] Data name information;

[0105] Data creation time information;

[0106] Data modification time information;

[0107] Database table name information;

[0108] Data instance name information;

[0109] Data editor information;

[0110] Data creator information;

[0111] Data size information;

[0112] Data storage path information.

[0113] The above-mentioned meta information may also be meta information of a file.

[0114] Step S102: Generate a meta-information table structure according to the meta-information of the data in the data source.

[0115] The data table in step S102 is composed of three parts: table name, fields in the table and records in the table. The table structure is to set the data table file name, determine which fields the data table contains, the field name, field type, width and other information of each field. The meta information table structure may include a meta information name field and a meta information content field. In this embodiment, the step S102 generates the meta information table structure specifically including:

[0116] Step S102-1: setting column information of the meta information table structure according to the meta information of the data in the data source;

[0117] Step S102-2: Generate the meta information table structure according to the column information.

[0118] The step S102-1 in setting the column information of the meta information table structure may be to classify the obtained meta information according to the type to which it belongs, and each type is the meta information content, and the type is the column name. For example, the column name may be a data name or a file name, the creation time of the data or file, etc., and the content of the column information is the specific content of the corresponding column name. For example, the file name (file_name) is ocr_11.txt, ocr_22.txt; the creation time (create_time) of ocr_11.txt is 2018-11-11, and the creation time (create_time) of ocr_22.txt is 2018-2-22.

[0119] The step S102 - 2 generates a meta information table structure according to the set column information, that is, records the meta information in the form of a data table.

[0120] Based on determining which data source the meta information is obtained from, so as to facilitate obtaining the relevant meta information and data content of the data source, the method may also include:

[0121] A meta information identifier is added to the column information of the set meta information table structure.

[0122] The meta information identifier can be represented by the keyword meta, and the identifier meta indicates that the meta information with the identifier needs to establish a data table fused with the meta information through the data table in the corresponding data source to perform operations such as data reading, insertion, or deletion. Figure 2 As shown, Figure 2 It is a schematic diagram of setting a meta information table structure for different source data in an embodiment of a data fusion method provided in the present application.

[0123] Figure 2When the data source is an object storage type, the data in the object storage database may include the following fields of meta information: bucket_name, file_name, create_time, last_access_time, last_update_time, file_length, etc. Meta can be added after each field of meta information. For example: the meta added to bucket_name indicates that the meta information is the corresponding storage (bucket) path in the object storage file; the meta added to file_name indicates that the meta information is the file name of the object storage, etc., that is:

[0124] bucket_name varchar(120)META comment'From the meta information, the object storage path corresponding to the object storage file';

[0125] file_name varchar(20)META comment'From the meta information, the object where the file is stored'.

[0126] When the data source is a table storage type, the data in the table storage database can include the following fields of meta information: instance_name, table_name, create_time, table_type, etc. Similarly, you can also add meta after each field of meta information. For example, the meta added to instance_name indicates that the meta information is the instance name; the meta added to table_name indicates that the meta information is the table name, etc., that is:

[0127] Instance_name varchar(20)META comment'From the meta information, the table where the instance is stored';

[0128] Table_name varchar(20) META comment 'From the meta information, the name of the table where the table is stored'.

[0129] The above meta information fields will not be described here one by one. For specific settings, please refer to Figure 2 shown.

[0130] In this embodiment, the meta information table structure may be generated at the database definition language (DDL) layer. The database definition language DDL may be mainly used to set or change the table structure, data type, connections and constraints between tables, and other operations.

[0131] Regardless of setting the meta information table structure or setting the column information of the meta information table structure, the meta information needs to be integrated into the data table describing the data content in the data source, which will be described in detail in step S103.

[0132] Step S103: Generate a data table integrated with the meta information according to the meta information table structure and a data table used to describe the data content in the data source.

[0133] The purpose of step S103 is to construct a data table including meta-information and data content by combining the meta-information table structure describing the data attributes of the data source and the data table used to describe the data content in the data source. In other words, the data table includes not only the data content but also the meta-information.

[0134] After setting the column information of the meta information table structure according to the meta information of the data in the data source, and generating the meta information table structure according to the column information, the step S103 may include:

[0135] According to the column information in the meta information table structure and the data table used to describe the data content in the data source, a data table merging the column information is generated.

[0136] After the meta information identifier is added to the column information in the meta information table structure, the step S103 may include:

[0137] Determining the data source corresponding to the meta-information according to the meta-information identifier in the column information;

[0138] A data table incorporating the meta information is generated according to the column information having the meta information identifier and the data table of the determined data content in the data source.

[0139] Whether generating a data table fused with the column information according to the column information in the meta information table structure and a data table used to describe the data content in the data source; or generating a data table fused with the meta information according to the column information with the meta information identifier and a data table of the determined data content in the data source, the specific generating method may include:

[0140] The meta information table structure is spliced ​​into a data table used to describe the data content in the data source to generate a data table integrated with the meta information.

[0141] When a data table fused with the column information is generated based on the column information in the meta information table structure and the data table used to describe the data content in the data source, the column information may be spliced ​​into the data table of the data content in the data source; when a data table fused with the meta information is generated based on the column information identified by the meta information and the data table used to describe the data content in the data source, the column information may be spliced ​​into the data table of the data content in the data source. The so-called splicing may be linking the column information in the meta information table structure to the data table of the data content to form a wider data table with the meta information after fusion, and the fused data table includes the meta information and the data content.

[0142] To ensure that the data source storage capacity of the data table remains unchanged after the metadata is merged, step S103 may include:

[0143] The meta information table structure is spliced ​​into a data table used to describe the data content in the data source to generate a data table integrated with virtual meta information.

[0144] That is, when the data table with the meta information is integrated to form a data table wider than the source data table, the meta information in the data table is virtual, and the user can see the meta information but cannot modify it; while the data content in the data table is real and the user can modify it.

[0145] It can be understood that generating a data table integrated with the meta information may also be generating a new data table, in which the new data table includes the meta information and the data content.

[0146] After the data table with the meta information is generated, step S103 can provide it to the computing operator on the upper layer of the data lake, so that the computing operator can analyze and process the data according to the data table with the meta information, and provide it to the client, thereby completing the real-time supply of data flow.

[0147] Based on the above content, the generation of the data table with the meta information can automatically splice the fields of the meta information into the data table describing the data content in the data source through the TableColumn Expander operator in the data lake. Therefore, it can also include:

[0148] A data table for describing the data content in the data source is obtained. Specifically, a data table with data content can be pulled from the data source through a Table Scan operator.

[0149] The above is a detailed description of an embodiment of a data fusion method provided by the present application. Corresponding to the aforementioned embodiment of a data fusion method, the present application also discloses an embodiment of a data fusion device. Figure 3 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0150] like Figure 3 As shown, Figure 3 : is a structural schematic diagram of an embodiment of a data fusion device provided by the present application. The device comprises:

[0151] The acquisition unit 301 is used to acquire meta information used to describe data attributes in a data source.

[0152] The acquisition unit 301 may specifically acquire meta information for describing data attributes in the data source according to document information provided by the data source.

[0153] The meta information may include at least the following information:

[0154] Data name information;

[0155] Data creation time information;

[0156] Data modification time information;

[0157] Database table name information;

[0158] Data instance name information;

[0159] Data editor information;

[0160] Data creator information;

[0161] Data size information;

[0162] Data path information.

[0163] The generating unit 302 is used to generate a meta information table structure according to the meta information of the data in the data source.

[0164] The generating unit 302 includes: a setting subunit and a generating subunit;

[0165] The setting subunit is used to set the column information of the meta information table structure according to the meta information of the data in the data source.

[0166] The generating subunit is used to generate the meta information table structure according to the column information.

[0167] The fusion unit 303 is used to generate a data table fused with the meta information according to the meta information table structure and a data table used to describe the data content in the data source.

[0168] The fusion unit 303 is specifically used to generate a data table fused with the meta information according to the column information of the meta information table structure and the data table used to describe the data content in the data source.

[0169] It also includes: an identification setting unit, which is used to add a meta information identification in the column information of the set meta information table structure.

[0170] Based on the identifier setting unit adding a meta information identifier to the column information of the set meta information table structure, the fusion unit 303 includes:

[0171] A data source determining subunit, configured to determine the data source corresponding to the meta information according to the meta information identifier in the column information;

[0172] The fusion unit 303 is specifically configured to generate a data table fused with the meta-information according to the data source corresponding to the meta-information determined in the data source determination subunit and the column information having the meta-information identifier.

[0173] The fusion unit 303 includes: a splicing subunit, which is used to splice the meta information table structure into a data table used to describe the data content in the data source, and generate a data table fused with the meta information.

[0174] The fusion unit 303 splices the meta information table structure into a data table used to describe the data content in the data source to generate a data table fused with virtual meta information.

[0175] When the fusion unit 303 fuses the meta information table structure into the data table of the data content in the data source, it is necessary to obtain the data table of the data content in the data source. Therefore, the device may further include:

[0176] The data table acquisition unit is used to acquire a data table for describing the data content in the data source. The specific acquisition method can refer to the description of step S103 in the above method, which will not be repeated here.

[0177] Based on the above content, the present application also provides a data processing method based on metadata, which is a method for processing operation requests for a database based on a data table fused with metadata. Therefore, the description of this method is only a summary description, and the specific content can refer to the above description of the data fusion method.

[0178] Please refer to Figure 4 As shown, Figure 4 : is a flowchart of an embodiment of a data processing method based on metadata provided by the present application, the method comprising:

[0179] Step S401: Obtain an operation request for a meta-information operation for describing data attributes.

[0180] In this embodiment, the operation request in step S401 may be an SQL-based operation request, including at least the following SQL operation request:

[0181] Data information query request, i.e. Query request;

[0182] Data information insertion request, i.e. Insert request;

[0183] Data information deletion request, i.e. Delete request;

[0184] Data information update request, i.e. Update request.

[0185] Of course, other SQL operation requests may also be included.

[0186] Step S402: according to the operation request, extracting data information satisfying the operation conditions in the operation request from the data table integrated with the meta-information.

[0187] The specific implementation process of S402 may include:

[0188] According to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, determining the meta information that satisfies the operation condition in the operation request;

[0189] Generate a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and a data table used to describe the data content in the data source;

[0190] In the generated data table incorporating the meta information, data information satisfying the operation condition in the operation request is extracted.

[0191] Based on the above content, it can be known that the data processing method based on meta information may also include:

[0192] Get the metadata used to describe data attributes in the data source;

[0193] Generate a meta information table structure according to the meta information of the data in the data source;

[0194] A data table incorporating the meta information is generated according to the meta information table structure and a data table used to describe the data content in the data source.

[0195] According to the metadata of the data in the data source, generating the metadata table structure may include:

[0196] Setting column information of the meta information table structure according to the meta information of the data in the data source;

[0197] The meta information table structure is generated according to the column information.

[0198] After the meta information table structure is generated based on the above setting of the column information of the meta information table structure, generating a data table incorporating the meta information may include:

[0199] According to the column information of the meta information table structure and the data table used to describe the data content in the data source, a data table merging the column information is generated.

[0200] In order to improve the speed of data processing, the specific implementation process of step S402 may include:

[0201] According to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, determining the meta information that satisfies the operation condition in the operation request;

[0202] Generate a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and a data table used to describe the data content in the data source;

[0203] In the generated data table incorporating the meta information, data information satisfying the operation condition in the operation request is extracted.

[0204] The operation condition may be the name of the meta-information required to be operated in the operation request. For example, when the operation request is a query request, the request content includes file_name of ocr_11.txt and create_time of 2018-11-11, where ocr_11.txt and 2018-11-11 are the operation conditions.

[0205] According to the operation request, in the meta-information table structure generated in advance according to the meta-information of the data in the data source, the meta-information that satisfies the operation conditions in the operation request is determined. It can be understood that the generated meta-information table structure includes multiple column information, and each column information corresponds to multiple meta-information fields under the column name, for example: the column names are file_name and create_time, etc., and the meta-information fields for the column name file_name are ocr_11.txt, ocr_22.txt, and so on. Of course, the meta-information fields for file_name are not limited to the above content. The meta-information fields for the column name create_time are 2018-11-11, 2018-11-22, etc. If in the operation request, the request content is file_name=ocr_11.txt, create_time=2018-11-11, then the meta-information that is the same as the request content is determined in the meta-information structure table, as shown in Table 1 below:

[0206] Table 1

[0207] file_name create_time …… ocr_11.txt 2018-11-11 …… ocr_11.txt 2018-11-11 …… ocr_22.txt 2018-11-22 …… ocr_22.txt 2018-11-22 ……

[0208] The data table after the meta information table structure is merged with the data table of the data content in the data source is shown in Table 2 below:

[0209] Table 2

[0210]

[0211]

[0212] The data table fused with the meta information is generated according to the determined meta information satisfying the operation conditions in the operation request and the data table used to describe the data content in the data source, which is to determine the meta information identical to the request content in the meta information structure table, for example: file_name=ocr_11.txt, create_time=2018-11-11; the data table used to describe the data content in the data source, for example: content is aaa, bbb, ccc; the two are spliced ​​together to form a fused data table with the meta information, as shown in Table 3 below:

[0213] content file_name create_time …… aaa ocr_11.txt 2018-11-11 …… bbb ocr_11.txt 2018-11-11 ……

[0214] That is, in the meta information table structure, the meta information field is determined according to the request content in the operation request, and the meta information field is spliced ​​into the data table in the form of columns to form a data table that is wider than the source data table. Therefore, the method may also include:

[0215] Setting column information of the meta information table structure according to the meta information of the data in the data source;

[0216] The meta information table structure is generated according to the column information.

[0217] The step of determining, according to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, meta information that satisfies the operation condition in the operation request includes:

[0218] In setting the column information of the meta information table structure according to the meta information of the data in the data source, determining the column information of the meta information table structure that satisfies the operation condition in the operation request;

[0219] The step of generating a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and the data table used to describe the data content in the data source comprises:

[0220] The column information of the meta information table structure determined to meet the operation condition in the operation request and the data table used to describe the data content in the data source are combined to generate a data table fused with the meta information.

[0221] Based on the above content, in the process of processing data for the operation request, the meta information may come from different data source types, that is, different data sources may provide meta information, such as: a data source of an object storage database, a data source of a cloud storage database, a data source of a table storage database, etc. Therefore, it is necessary to determine which data source the meta information comes from according to the request content in the operation request. Therefore, the data processing method based on meta information may also include:

[0222] Adding a meta information identifier to the column information of the set meta information table structure;

[0223] The step of generating a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and the data table used to describe the data content in the data source comprises:

[0224] Determining the data source according to the meta information identifier in the column information;

[0225] A data table incorporating the meta information is generated according to the column information having the meta information identifier and the data table of the determined data content in the data source.

[0226] After the step S402 extracts the data information that meets the operation condition in the operation request, the step S403 is executed.

[0227] Step S403: Send data information that meets the operating conditions.

[0228] The purpose of step 403 is to output the data information that meets the operation conditions and provide it to the computing operator of the data lake to complete the real-time supply of data streams so that users can obtain data information according to operation requests.

[0229] The above is a description of an embodiment of a data processing method based on metadata provided by the present application. For some contents of the method, reference can be made to the description of the above data fusion method embodiment, so some repeated contents are omitted. Corresponding to the embodiment of the data processing method based on metadata, the present application also discloses an embodiment of a data processing device based on metadata, please refer to Figure 5 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0230] like Figure 5 As shown, Figure 5 1 is a schematic diagram of a data processing device embodiment based on metadata provided by the present application. The device comprises:

[0231] The acquisition unit 501 is used to acquire an operation request including meta-information for describing data attributes.

[0232] The operation request in the acquisition unit 501 may include at least one of the following operation requests:

[0233] Data information query request;

[0234] Data information insertion request;

[0235] Request for deletion of data information;

[0236] Data information update request.

[0237] An extraction unit 502 is used to extract data information that meets the operation condition in the operation request from the data table integrated with the meta information according to the operation request;

[0238] The extraction unit 502 includes:

[0239] A determination subunit, configured to determine, according to the operation request, in a meta-information table structure generated in advance according to the meta-information of the data in the data source, meta-information that satisfies the operation condition in the operation request;

[0240] The sending unit 503 is used to send data information that meets the operating conditions.

[0241] It also includes: a meta information acquisition unit, a meta information table structure generation unit and a data table generation unit.

[0242] The meta information acquisition unit is used to acquire meta information used to describe data attributes in the data source.

[0243] The meta information table structure generating unit is used to generate a meta information table structure according to the meta information of the data in the data source.

[0244] The data table generating unit is used to generate a data table integrating the meta information according to the meta information table structure and a data table used to describe the data content in the data source.

[0245] Wherein, the meta information table structure generating unit comprises:

[0246] The setting subunit is used to set the column information of the meta information table structure according to the meta information of the data in the data source.

[0247] A generating subunit is used to generate the meta information table structure according to the column information.

[0248] The data table generating unit is specifically configured to generate a data table incorporating the column information according to the column information of the meta information table structure and a data table for describing the data content in the data source.

[0249] To speed up processing, also include:

[0250] a determining unit, configured to determine, according to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, meta information that satisfies the operation condition in the operation request;

[0251] The data table generating unit is used to generate a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and the data table used to describe the data content in the data source.

[0252] The extraction unit 502 is specifically configured to extract data information that meets the operation condition in the operation request from the generated data table incorporating the meta information.

[0253] In another embodiment, the data processing method based on metadata may further include:

[0254] A column information setting unit, used to set the column information of the meta information table structure according to the meta information of the data in the data source;

[0255] A meta information table structure generating unit, used for generating the meta information table structure according to the column information;

[0256] The determining unit is specifically configured to determine, among the column information of the meta information table structure set according to the meta information of the data in the data source, the column information of the meta information table structure that satisfies the operation condition in the operation request;

[0257] The data table generating unit is specifically used to generate a data table incorporating the meta information by combining the column information of the meta information table structure determined to meet the operation condition in the operation request and the data table used to describe the data content in the data source.

[0258] The data processing method based on meta information further includes:

[0259] An identifier adding unit, used for adding a meta information identifier to the column information of the set meta information table structure;

[0260] The data table generating unit comprises:

[0261] A data source determination subunit, configured to determine the data source according to the meta-information identifier in the column information;

[0262] The data generation subunit is used to generate a data table integrated with the meta information according to the column information with the meta information identifier and the data table of the data content in the determined data source.

[0263] The above is an overview of an embodiment of a data processing device based on metadata provided in the present application. For specific contents, please refer to the description of the embodiment of the data processing method based on metadata and the description of the data fusion method, so they will not be repeated here.

[0264] Based on the above content, this application also provides a data processing method, please refer to Figure 6 As shown, the method includes:

[0265] Step S601: The computing node obtains an operation request from the front-end node for meta-information used to describe data attributes;

[0266] In step S601, the front-end node (Frontnode) receives the operation request from the client, so the data lake can analyze the big data. Therefore, the front-end node can include multiple, i.e., the front-end node cluster (Frontnodepool). The so-called data lake is a data analysis service system based on big data. When doing analysis based on the data lake, there is no need to do any ETL (the ETL is used to describe the process of extracting, transforming, and loading data from the source to the destination), data migration and other pre-processes, so as to realize the correlation analysis of big data across various heterogeneous data sources, thereby greatly saving costs and improving user experience.

[0267] To ensure load balancing, the operation request is distributed to different front-end nodes through the load balancing server (LoadBalancer) between the front-end node and the client (APP). The front-end node parses the received operation request and generates a data table with metadata based on the parsed structure through DDL optimization. Specifically, when the Table Scan operator in the data lake pulls the specific data part from the data source, the Table ColumnExpander operator automatically splices the metadata fields that the user needs to use in SQL into the data row pulled by the Table Scan, forming a wider new data stream with complete fields, which is provided to the calculation operator in the data. The calculation operator will perform SQL calculations based on the data table with metadata to complete the real-time supply of data streams.

[0268] Step S602: the computing node extracts data information satisfying the operation conditions in the operation request from the data table integrated with the meta-information in response to the operation request for describing the meta-information of data attributes.

[0269] The specific operation process of step S602 may refer to the description of step S103 and step S402 above.

[0270] Step S603: the computing node sends the extracted data information satisfying the operation condition in the operation request to the front-end node.

[0271] In step S603, after the computing node performs calculations based on the data information provided by the front-end node, the data information is sent to the client, thereby completing the real-time supply of the data stream.

[0272] The above is an overview of an embodiment of a data processing method provided in this application. The specific application scenario may be the data processing process performed when performing data analysis based on a data lake. The specific content of the process can refer to the description of the above-mentioned data fusion method and the data processing method based on metadata, which will not be repeated here.

[0273] Based on the above content, the present application also provides a computer storage medium for storing a program;

[0274] When the program is read and executed by the processor, the processor executes the steps in the data fusion method described above, or the steps in the data processing method based on metadata described above, or the steps in the data lake data processing method described above.

[0275] The present application also provides an electronic device, comprising:

[0276] processor;

[0277] A memory for storing a program, which, when read and executed by the processor, enables the electronic device to execute the steps in the data fusion method as described above, or the steps in the data processing method based on metadata as described above, or the steps in the data processing method based on a data lake as described above.

[0278] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0279] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0280] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0281] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0282] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A data fusion method, characterized in that: include: Get the metadata used to describe data attributes in the data source; Generate a meta information table structure according to the meta information of the data in the data source; The meta information table structure is spliced ​​into a data table used to describe the data content in the data source to generate a data table integrated with the meta information.

2. The data fusion method according to claim 1, characterized in that: The step of generating a meta information table structure according to the meta information of the data in the data source includes: Setting column information of the meta information table structure according to the meta information of the data in the data source; The meta information table structure is generated according to the column information.

3. The data fusion method according to claim 2, characterized in that: The step of splicing the meta information table structure into a data table used to describe the data content in the data source to generate a data table incorporating the meta information includes: The column information of the meta information table structure is spliced ​​into a data table used to describe the data content in the data source to generate a data table integrated with the meta information.

4. The data fusion method according to claim 2, characterized in that: Also includes: Adding a meta information identifier to the column information of the set meta information table structure; The step of splicing the meta information table structure into a data table used to describe the data content in the data source to generate a data table incorporating the meta information includes: Determining the data source corresponding to the meta-information according to the meta-information identifier in the column information; A data table incorporating the meta information is generated according to the column information having the meta information identifier and the data table of the determined data content in the data source.

5. The data fusion method according to claim 1, characterized in that: The step of splicing the meta information table structure into a data table used to describe the data content in the data source to generate a data table incorporating the meta information includes: The meta information table structure is spliced ​​into a data table used to describe the data content in the data source to generate a data table integrated with virtual meta information.

6. The data fusion method according to claim 1, characterized in that: The step of obtaining meta information from the data source for describing data attributes includes: According to the document information provided by the data source, obtain the meta information used to describe the data attributes in the data source.

7. The data fusion method according to claim 1, characterized in that: The meta information includes at least the following information: Data name information; Data creation time information; Data modification time information; Database table name information; Data instance name information; Data editor information; Data creator information; Data size information; Data path information.

8. The data fusion method according to claim 1, characterized in that: Also includes: Get a data table for describing the data content in the data source.

9. A data fusion device, characterized in that: include: An acquisition unit, used for acquiring meta information used to describe data attributes in a data source; A generating unit, used for generating a meta information table structure according to the meta information of the data in the data source; The fusion unit is used to splice the meta information table structure into a data table used to describe the data content in the data source to generate a data table fused with the meta information.

10. A data processing method based on meta information, characterized in that: include: Get an operation request for a meta-information operation used to describe data attributes; According to the operation request, extracting data information that meets the operation condition in the operation request from a data table integrated with the meta information, wherein the data table integrated with the meta information is generated by splicing a meta information table structure into a data table used to describe data content in a data source, and the meta information table structure is generated according to the meta information used to describe data attributes in the data source; Send data information that meets the operating conditions.

11. The data processing method based on meta information according to claim 10, characterized in that: Also includes: Get the metadata used to describe data attributes in the data source; Generate a meta information table structure according to the meta information of the data in the data source; A data table incorporating the meta information is generated according to the meta information table structure and a data table used to describe the data content in the data source.

12. The data processing method based on meta information according to claim 11, characterized in that: The step of generating a meta information table structure according to the meta information of the data in the data source includes: Setting column information of the meta information table structure according to the meta information of the data in the data source; The meta information table structure is generated according to the column information.

13. The data processing method based on meta information according to claim 12, characterized in that: The step of generating a data table incorporating the meta information according to the meta information table structure and a data table for describing the data content in the data source comprises: According to the column information of the meta information table structure and the data table used to describe the data content in the data source, a data table merging the column information is generated.

14. The data processing method based on meta information according to claim 10, characterized in that: Also includes: According to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, determining the meta information that satisfies the operation condition in the operation request; Generate a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and a data table used to describe the data content in the data source; The extracting, according to the operation request, data information satisfying the operation condition in the operation request from the data table integrated with the meta information, comprises: In the generated data table incorporating the meta information, data information satisfying the operation condition in the operation request is extracted.

15. The data processing method based on meta information according to claim 14, characterized in that: Also includes: Setting column information of the meta information table structure according to the meta information of the data in the data source; Generate the meta information table structure according to the column information; The step of determining, according to the operation request, in a meta information table structure generated in advance according to the meta information of the data in the data source, meta information that satisfies the operation condition in the operation request includes: In setting the column information of the meta information table structure according to the meta information of the data in the data source, determining the column information of the meta information table structure that satisfies the operation condition in the operation request; The step of generating a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and the data table used to describe the data content in the data source comprises: The column information of the meta information table structure determined to meet the operation condition in the operation request and the data table used to describe the data content in the data source are combined to generate a data table fused with the meta information.

16. The data processing method based on meta information according to claim 15, characterized in that: Also includes: Adding a meta information identifier to the column information of the set meta information table structure; The step of generating a data table incorporating the meta information according to the determined meta information satisfying the operation condition in the operation request and the data table used to describe the data content in the data source comprises: Determining the data source according to the meta information identifier in the column information; A data table incorporating the meta information is generated according to the column information having the meta information identifier and the data table of the determined data content in the data source.

17. A data processing device based on meta-information, characterized in that: include: An acquisition unit, used for acquiring an operation request including meta information for describing data attributes; an extraction unit, configured to extract, according to the operation request, data information satisfying the operation condition in the operation request from a data table integrated with the meta information, wherein the data table integrated with the meta information is generated by splicing a meta information table structure into a data table used to describe data content in a data source, and the meta information table structure is generated according to the meta information used to describe data attributes in the data source; The sending unit is used to send data information that meets the operating conditions.

18. A data processing method, characterized in that: include: The computing node obtains an operation request from the front-end node for meta-information used to describe data attributes; The computing node extracts data information that meets the operation conditions in the operation request from a data table integrated with the meta-information for an operation request for describing the meta-information of data attributes, wherein the data table integrated with the meta-information is generated by splicing a meta-information table structure into a data table for describing the data content in a data source, and the meta-information table structure is generated according to the meta-information in the data source for describing the data attributes; The computing node sends the extracted data information satisfying the operation condition in the operation request to the front-end node.

19. A computer storage medium for storing programs; When the program is read and executed by a processor, the processor executes the steps in the data fusion method described in any one of claims 1 to 8, or the steps in the metadata-based data processing method described in any one of claims 10 to 16, or the steps in the data processing method described in claim 18.

20. An electronic device, comprising: processor; A memory for storing a program, wherein when the program is read and executed by the processor, the electronic device executes the steps in the data fusion method according to any one of claims 1 to 8, the steps in the data processing method based on metadata according to any one of claims 10 to 16, or the steps in the data processing method according to claim 18.

Images