Data processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a data processing method and device, electronic equipment and computer readable storage medium, and can be applied to the scene of analyzing the attributes of the data warehouse data table. The method comprises: obtaining a mirror file corresponding to original data in a distributed file system, and obtaining initial metadata corresponding to the original data; creating a mirror file metadata table and a metadata result table corresponding to the mirror file and the initial metadata respectively; performing merging and parsing processing based on the temporary tables corresponding to the mirror file metadata table and the metadata result table respectively to generate an attribute statistical result table of a business attribute corresponding to the original data; and performing data screening processing on the attribute statistical result table to obtain target attribute statistical data corresponding to the business attribute. The present disclosure can integrate the mirror file in the distributed file system and the metadata in the relational database to statistically analyze the multi-dimensional business attributes of the original data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data operation and maintenance, in particular, to a data processing method, a data processing apparatus, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation and loading, which is a mechanism that can store, query and analyze large-scale data stored in Hadoop. As one of the main technical solutions of data warehouse, the SQL-like processing mode provided by Hive has been widely applied to various big data offline processing systems.

[0003] Since Hive data is stored in the traditional Hadoop Distributed File System (HDFS), unreasonable use of Hive may affect the performance of the big data cluster, such as Hive table Block block size problem, Hive table small file problem, and the like. At the same time, Hive data warehouse operation and maintenance personnel also urgently need to know the storage status of the current Hive table in all directions, so as to expose unknown hidden troubles in advance.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present disclosure is to provide a data processing method, a data processing apparatus, an electronic device and a computer readable storage medium, thereby at least to some extent overcoming the problem that the existing data warehouse implementation scheme lacks effective solutions for business table attribute statistics operation and maintenance of Hive itself.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a data processing method is provided, comprising: obtaining a mirror file corresponding to original data in a distributed file system, and obtaining initial metadata corresponding to the original data; creating a mirror file metadata table and a metadata result table corresponding to the mirror file and the initial metadata respectively; performing merge analysis processing based on temporary tables corresponding to the mirror file metadata table and the metadata result table respectively, to generate an attribute statistics result table of a business attribute corresponding to the original data; and performing data screening processing on the attribute statistics result table to obtain target attribute statistics data corresponding to the business attribute.

[0008] In an example embodiment of the present disclosure, the obtaining the mirror image file corresponding to the original data in the distributed file system comprises: in response to a mirror image file extraction instruction, obtaining the mirror image file corresponding to the original data from the distributed file system; performing file parsing processing on the mirror image file to obtain a parsed mirror image file; and storing the parsed mirror image file to a first storage path of the distributed file system.

[0009] In an example embodiment of the present disclosure, the obtaining the initial metadata corresponding to the original data comprises: connecting an execution end of a distributed file system to a metadata repository of the original data; generating a remote interaction instruction based on the execution end, obtaining the initial metadata from the metadata repository based on the remote interaction instruction; and storing the initial metadata to a second storage path of the distributed file system.

[0010] In an example embodiment of the present disclosure, the obtaining the initial metadata from the metadata repository based on the remote interaction instruction comprises: obtaining a pre-constructed query script, obtaining an initial metadata table from the metadata repository; performing parsing processing on the initial metadata table based on the query script to obtain a metadata result table; obtaining a data extraction component, and storing the metadata result table to a second storage path of the distributed file system as the initial metadata by using the data extraction component.

[0011] In an example embodiment of the present disclosure, the respectively creating a mirror image file metadata table and a metadata result table corresponding to the mirror image file and the initial metadata respectively comprises: creating a metadata table corresponding to the mirror image file in a data warehouse as the mirror image file metadata table; creating a metadata table corresponding to the initial metadata in the data warehouse as the metadata result table; and taking the mirror image file metadata table and the metadata result table as external tables of data files in the distributed file system.

[0012] In an example embodiment of the present disclosure, the performing merging and parsing processing based on the temporary tables corresponding to the mirror image file metadata table and the metadata result table respectively to generate an attribute statistical result table of a business attribute corresponding to the original data comprises: respectively creating a mirror image file temporary table and a metadata temporary table corresponding to the mirror image file metadata table and the metadata result table respectively; performing merging processing on the mirror image file temporary table and the metadata temporary table to generate a merged data table; and performing parsing processing on the merged data table to generate the attribute statistical result table.

[0013] In an example embodiment of the present disclosure, the creating the mirror file metadata table and the metadata result table respectively, and the corresponding mirror file temporary table and the metadata temporary table respectively, includes: performing aggregate grouping processing on the mirror file metadata table to obtain the mirror file temporary table corresponding to the mirror file metadata table; and performing partition aggregate processing on the metadata result table to obtain the metadata temporary table corresponding to the metadata result table.

[0014] In an example embodiment of the present disclosure, the aggregate grouping processing on the mirror file metadata table to obtain the mirror file temporary table corresponding to the mirror file metadata table includes: obtaining original table information of the mirror file metadata table, performing regular matching processing on the original table information to obtain valid table information of the mirror file metadata table; performing statistical processing on the number of files in the mirror file metadata table to obtain file statistical information corresponding to the mirror file metadata table; and generating the mirror file temporary table based on the valid table information and the file statistical information.

[0015] In an example embodiment of the present disclosure, the merging processing on the mirror file temporary table and the metadata temporary table to generate a merged data table includes: determining a first table field corresponding to the mirror file temporary table; determining a second table field corresponding to the metadata temporary table; and performing merging processing on the mirror file temporary table and the metadata temporary table based on the first table field and the second table field to obtain the merged data table.

[0016] In an example embodiment of the present disclosure, the data filtering processing on the attribute statistical result table to obtain target attribute statistical data corresponding to the business attribute includes: obtaining a merged data table generated based on the mirror file temporary table and the metadata temporary table; performing merging processing on the merged data table and the attribute statistical result table to generate an initial statistical result table; obtaining a pre-configured attribute sorting index, and performing sorting processing on attribute data of the initial statistical result table based on the attribute sorting index to obtain the target attribute statistical data.

[0017] In an example embodiment of the present disclosure, the above method further includes: obtaining a pre-configured template engine; extracting to-be-displayed data from the target attribute statistical data by using the template engine; sending the to-be-displayed data to a visual execution end, and displaying the to-be-displayed data through a visual interface of the visual execution end.

[0018] In an example embodiment of the present disclosure, the method further includes: obtaining a pre-configured content intercepting component; intercepting, by the content intercepting component, page content in the visualization interface to obtain alarm auxiliary data; and generating alarm information based on the alarm auxiliary data.

[0019] According to a second aspect of the present disclosure, a data processing apparatus is provided, including: a data acquisition module, configured to acquire a mirror file corresponding to original data in a distributed file system, and acquire initial metadata corresponding to the original data; a metadata table creation module, configured to create a mirror file metadata table and a metadata result table corresponding to the mirror file and the initial metadata, respectively; a merging processing module, configured to perform merging and parsing processing based on temporary tables corresponding to the mirror file metadata table and the metadata result table, respectively, to generate an attribute statistical result table of a business attribute corresponding to the original data; and a data screening module, configured to perform data screening processing on the attribute statistical result table to obtain target attribute statistical data corresponding to the business attribute.

[0020] In an example embodiment of the present disclosure, the data acquisition module includes a mirror file acquisition unit, configured to: in response to a mirror file extraction instruction, acquire the mirror file corresponding to the original data from the distributed file system; perform file parsing processing on the mirror file to obtain a parsed mirror file; and store the parsed mirror file to a first storage path of the distributed file system.

[0021] In an example embodiment of the present disclosure, the data acquisition module includes a metadata acquisition unit, configured to: connect an execution end of a distributed file system to a metadata repository of the original data; generate a remote interaction instruction based on the execution end, acquire the initial metadata from the metadata repository based on the remote interaction instruction; and store the initial metadata to a second storage path of the distributed file system.

[0022] In an example embodiment of the present disclosure, the metadata acquisition unit includes a metadata acquisition subunit, configured to: acquire a pre-constructed query script, acquire an initial metadata table from the metadata repository; perform parsing processing on the initial metadata table based on the query script to obtain a metadata result table; acquire a data extraction component, and store the metadata result table to a second storage path of the distributed file system as the initial metadata by using the data extraction component.

[0023] In an example embodiment of the present disclosure, the metadata creation module comprises a metadata creation unit configured to: create a metadata table corresponding to the mirror file in the data warehouse as the mirror file metadata table; create a metadata table corresponding to the initial metadata in the data warehouse as the metadata result table; and set the mirror file metadata table and the metadata result table as external tables of data files in the distributed file system.

[0024] In an example embodiment of the present disclosure, the merging processing module comprises a merging processing unit configured to: create a mirror file temporary table and a metadata temporary table respectively corresponding to the mirror file metadata table and the metadata result table; perform merging processing on the mirror file temporary table and the metadata temporary table to generate a merged data table; and perform parsing processing on the merged data table to generate the attribute statistics result table.

[0025] In an example embodiment of the present disclosure, the merging processing unit comprises a temporary table creation unit configured to: perform aggregate grouping processing on the mirror file metadata table to obtain a mirror file temporary table corresponding to the mirror file metadata table; and perform partition aggregate processing on the metadata result table to obtain a metadata temporary table corresponding to the metadata result table.

[0026] In an example embodiment of the present disclosure, the temporary table creation unit comprises a temporary table creation sub-unit configured to: obtain original table information of the mirror file metadata table, perform regular matching processing on the original table information to obtain valid table information of the mirror file metadata table; perform statistical processing on the number of files in the mirror file metadata table to obtain file statistical information corresponding to the mirror file metadata table; and generate the mirror file temporary table based on the valid table information and the file statistical information.

[0027] In an example embodiment of the present disclosure, the merging processing unit comprises a merging processing sub-unit configured to: determine a first table field corresponding to the mirror file temporary table; determine a second table field corresponding to the metadata temporary table; and perform merging processing on the mirror file temporary table and the metadata temporary table based on the first table field and the second table field to obtain the merged data table.

[0028] In an example embodiment of the present disclosure, the data filtering module comprises a data filtering unit configured to: obtain a merged data table generated based on the mirror file temporary table and the metadata temporary table; perform merging processing on the merged data table and the attribute statistics result table to generate an initial statistics result table; obtain a pre-configured attribute sorting index; perform sorting processing on attribute data of the initial statistics result table based on the attribute sorting index to obtain the target attribute statistical data.

[0029] In an example embodiment of the present disclosure, the data processing apparatus further comprises a data display module configured to: acquire a pre-configured template engine; extract data to be displayed from the target attribute statistical data by using the template engine; and send the data to be displayed to a visualization execution end, and display the data to be displayed through a visualization interface of the visualization execution end.

[0030] In an example embodiment of the present disclosure, the data processing apparatus further comprises an alarm information generation module configured to: acquire a pre-configured content intercepting component; intercept page content in the visualization interface by using the content intercepting component to obtain alarm auxiliary data; and generate alarm information based on the alarm auxiliary data.

[0031] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the data processing method according to any one of the preceding aspects.

[0032] According to a fourth aspect of the present disclosure, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement the data processing method according to any one of the preceding aspects.

[0033] The technical solutions provided by the present disclosure can include the following beneficial effects:

[0034] The data processing method in the example embodiments of the present disclosure, on the one hand, provides a scheme of analyzing attribute information of a business table in a data warehouse by integrating a mirror file and metadata, so as to avoid missing some index information caused by analyzing a single file. On the other hand, by using the temporary tables of the mirror file analysis table and the metadata analysis table to perform data merging and filtering processing, the original data can be fully analyzed, and the credibility of the final attribute statistical result data can be improved.

[0035] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. In the drawings:

[0037] Figure 1A schematic block diagram of a system architecture of an example application scenario according to some embodiments of the present disclosure is shown schematically;

[0038] Figure 2 A flowchart of a data processing method according to an example embodiment of the present disclosure is shown schematically;

[0039] Figure 3 A data flow diagram of storing a mirror file and initial metadata in a distributed file system according to an example embodiment of the present disclosure is shown schematically;

[0040] Figure 4 A flowchart of performing business attribute analysis by integrating a mirror file of raw data and initial metadata according to an example embodiment of the present disclosure is shown schematically;

[0041] Figure 5 A block diagram of a data processing apparatus according to an example embodiment of the present disclosure is shown schematically;

[0042] Figure 6 A block diagram of an electronic device according to an example embodiment of the present disclosure is shown schematically;

[0043] Figure 7 A schematic diagram of a computer readable storage medium according to an example embodiment of the present disclosure is shown schematically. DETAILED DESCRIPTION

[0044] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the several views.

[0045] Moreover, described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the techniques described can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail in order to avoid obscuring aspects of the disclosure.

[0046] The block diagrams illustrated in the drawings are merely functional entities and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in the form of software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0047] In the production and operation process of various industries, a large amount of related data will be generated, and the generated data can be used for subsequent data analysis. For example, the large amount of technical data generated by different processes in the semiconductor industry is basically landed in the Hive data warehouse, but there is no mature technical solution to quickly obtain the various complete storage attribute information of the Hive table. Although all the metadata information of the Hive table is stored in a relational database, such as an Oracle database, a MySQL database, etc., and HDFS also provides a variety of statistical commands, manual statistics based on thousands of tables is unrealistic and the most inefficient.

[0048] In addition, continuous data protection (CDP) adds more advanced management, performance tuning, key management and other novel features, and the Hive version is also constantly updated, but the focus of the high version of Hive is performance improvement, such as the four elements of ACID for correct execution of database transactions by default, including atomicity (atomicity, also known as indivisibility), consistency, isolation (independence), durability, replacement of execution engine, opening of low-latency analysis processing (Live long and process, LLAP) support, etc. A mature technical solution is still lacking in terms of Hive itself business table attribute statistics operation and maintenance.

[0049] Therefore, a data processing method and device, an electronic device and a computer readable storage medium are disclosed. First, reference is made to Figure 1 , Figure 1 A schematic block diagram of a system architecture of an exemplary application scenario of a data processing method and device to which the embodiments of the present disclosure can be applied is shown.

[0050] As Figure 1As shown, the system architecture 100 can include a distributed file storage system 101, a relational database 102, a server 103, a network 104, and one or more of visual execution ends 105, 106, 107. The network 104 serves as a medium to provide communication links between the visual execution ends 105, 106, 107 and the server 103. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, and the like. The visual execution ends 105, 106, 107 can be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, and tablet computers, and the like. It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above-mentioned system architecture 100 is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, the server 103 can be a server cluster composed of multiple servers.

[0051] The data processing method provided by the embodiments of the present disclosure is generally executed by the server 103, and accordingly, the data processing apparatus is generally provided in the server 103. However, it is easily understood by those skilled in the art that the data processing method provided by the embodiments of the present disclosure can also be executed by a terminal device, and accordingly, the data processing apparatus can also be provided in a terminal device, which is not specially limited in the present exemplary embodiment. For example, in an exemplary embodiment, the mirror file and the initial metadata corresponding to the original data can be obtained from the distributed file storage system 101 and the relational database 102, the server 103 processes the mirror file and the initial metadata by using the data processing method provided by the embodiments of the present disclosure to obtain target attribute statistical data corresponding to the original data, and the obtained target attribute statistical data is transmitted to the visual execution ends 105, 106, 107 through the network 104, so that the terminal device visual execution ends 105, 106, 107 display the to-be-displayed data in the target attribute statistical data.

[0052] In the present exemplary embodiment, first, a data processing method is provided, which can be implemented by using a server or a terminal device. In the present disclosure, the terminal can include mobile terminals such as mobile phones, tablet computers, notebook computers, palmtop computers, personal digital assistants (PDAs), and the like, and fixed terminals such as desktop computers. Figure 2 A schematic diagram of a data processing method flow according to some embodiments of the present disclosure is schematically shown. Referring to Figure 2 The data processing method can include the following steps:

[0053] Step S210, obtaining the mirror file corresponding to the original data in the distributed file system, and obtaining the initial metadata corresponding to the original data.

[0054] According to some example embodiments of the present disclosure, the original data can be original business data stored in the distributed file system. The mirror file can be a copy file of the original data in the distributed file system. The initial metadata can be data used to describe the original data, and the initial metadata is mainly information describing the data properties of the original data, which is used to support functions such as indicating the storage location of the original data, historical data, resource lookup, file recording, etc.

[0055] In various business scenarios of production and life, a large amount of technical data and user data can be generated. The above data is usually a large amount of data set that is difficult to collect, process and analyze. For example, in the semiconductor production and manufacturing scenario, the massive data generated in the production and manufacturing process can be stored in HDFS. Before data processing, the original data can be obtained from the distributed file system (such as HDFS), for example, the mirror file (fsimage file) corresponding to the original data can be obtained.

[0056] In addition, for the original data stored in HDFS, there is usually data describing its data characteristics, i.e. the initial metadata corresponding to the original data. The initial metadata is mainly information describing the data properties of the original data, which is information about the organization of the data, the domain of the data and their relationship. For the initial metadata of the original data, according to the storage characteristics of the data, it is known that the initial metadata is usually stored in a relational database, so the initial metadata can be obtained from the relational database, and the obtained mirror file and initial metadata are stored in a specified storage path.

[0057] Step S220, respectively creating a mirror file metadata table and a metadata result table corresponding to the mirror file and the initial metadata.

[0058] According to some example embodiments of the present disclosure, the mirror file metadata table can be a data table corresponding to the metadata of the mirror file created in the data warehouse. The metadata result table can be a data table corresponding to the data describing the initial metadata created in the data warehouse.

[0059] After storing the obtained mirror file and initial metadata of the original data in the specified storage path, metadata tables can be created to describe the above data, respectively. Specifically, a metadata table corresponding to the mirror file can be created in the data warehouse, and a metadata result table corresponding to the initial metadata can be created in the data warehouse.

[0060] Step S230, based on the mirror file metadata table and the metadata result table respectively corresponding to the temporary table, the merging and parsing processing is performed to generate the attribute statistical result table of the business attribute corresponding to the original data.

[0061] According to some example embodiments of the present disclosure, the temporary table can be the mirror file metadata table and the metadata result table established in the system temporary folder. The merging and parsing processing can be the merging processing of the related fields in the mirror file metadata table and the metadata result table, and the processing operation of information statistics and the like. The business attribute can be the attribute related to the original data stored in the data warehouse. The attribute statistical result table can be the data table composed of the related statistical information of the business attribute of the original data.

[0062] After the mirror file metadata table and the metadata result table corresponding to the mirror file and the initial metadata are created, in order to fully parse the original data, the mirror file metadata table and the metadata result table can be respectively parsed according to the analysis requirements to obtain the respective corresponding temporary table, so as to take the obtained temporary table as the data basis for subsequent statistical analysis of the business attribute.

[0063] After obtaining the temporary table corresponding to the mirror file metadata table and the metadata result table, the two temporary tables can be merged and parsed, for example, the two temporary tables are merged based on the fields of the table, the attribute information contained in the table is integrated, and the attribute statistical result table of the business attribute corresponding to the original data is generated.

[0064] Step S240, data filtering processing is performed on the attribute statistical result table to obtain the target attribute statistical data corresponding to the business attribute.

[0065] According to some example embodiments of the present disclosure, the data filtering processing can be the processing operation of filtering out the attribute related information meeting the requirements from the attribute statistical result table according to the pre-configured filtering rule. The target attribute statistical data can be the attribute statistical data meeting the requirements matched based on the filtering rule.

[0066] After obtaining the attribute statistical result table, the attribute statistical result table can be filtered according to the actual requirements, for example, the user can customize the attribute sorting index, and the business attribute data ranked in the top N positions is filtered from the attribute statistical result table according to the customized attribute sorting index as the target attribute statistical data corresponding to the business attribute, so as to obtain the attribute statistical data that the user wants to obtain.

[0067] According to the data processing method in the example embodiment, on one hand, a scheme is provided for analyzing attribute information of a business table in a data warehouse by integrating a mirror file and metadata, so that information omission of some indicators caused by analyzing a single file is avoided. On the other hand, through data merging and filtering processing of the temporary table of the mirror file analysis table and the metadata analysis table, the original data can be fully analyzed, and the reliability of the final attribute statistical result data is improved.

[0068] In the following, the data processing method in the example embodiment will be further described.

[0069] In an example embodiment of the disclosure, for step S210, the mirror file corresponding to the original data in the distributed file system is acquired, including: in response to a mirror file extraction instruction, acquiring the mirror file corresponding to the original data from the distributed file system; performing file analysis processing on the mirror file to obtain an analyzed mirror file; and storing the analyzed mirror file to a first storage path of the distributed file system.

[0070] The mirror file extraction instruction can be an operation instruction for acquiring the mirror file from the distributed file system. The file analysis processing can be a processing operation of converting the mirror file from an original data format to a specified data format. The analyzed mirror file can be an analyzed file obtained by performing file analysis processing on the mirror file. The first storage path can be a storage path in the distributed file system for storing the analyzed mirror file.

[0071] Reference Figure 3 For the original data stored in the HDFS, the mirror file corresponding to the original data can be acquired from the HDFS distributed storage system. For example, the mirror file can be acquired from the HDFS distributed storage system based on a mirror file extraction instruction. A related technical person can use the mirror file extraction instruction (such as HDFS fetchImage) to download the mirror file 310 from the HDFS distributed storage system to the server offline, and the mirror file acquired from the HDFS distributed storage system can be stored in the execution server in binary form, i.e., a binary file 320 stored in the execution server.

[0072] For the acquired binary file 320, the binary file 320 can be subjected to file analysis processing to obtain a corresponding readable file. For example, the mirror file is converted into a readable file using an oiv command. After the mirror file is analyzed using the oiv command, a corresponding comma-separated value (CSV, sometimes also referred to as character-separated value, because the separation character can also not be a comma) file 330 can be obtained. The CSV file stores tabular data (numbers and text) in plain text form.

[0073] In other exemplary embodiments of the present disclosure, the image file obtained from the HDFS distributed storage system can also be converted into other readable file formats, and the image file after format conversion is uploaded to the HDFS distributed storage system. The present disclosure does not make any special limitation on the specific file format obtained after conversion.

[0074] For the obtained CSV file 330, it can be stored as a parsed image file 340 in the distributed file system. Specifically, the parsed image file can be stored in the first storage path of the HDFS distributed storage system, such as the first storage path of the HDFS can be / user / xxx / tb_hdfs_metadata. Through the above steps, the image file of the original data can be obtained from the HDFS and stored in the HDFS as the data basis for subsequent data service attribute analysis.

[0075] In an exemplary embodiment of the present disclosure, for step S110, obtaining initial metadata corresponding to the original data includes: connecting an execution end of a distributed file system to a metadata repository of the original data; generating a remote interaction instruction based on the execution end, obtaining the initial metadata from the metadata repository based on the remote interaction instruction; and storing the initial metadata in a second storage path of the distributed file system.

[0076] The execution end of the distributed file system can be an execution operation end connected to the distributed file system, and the execution end can run a query script. The metadata repository can be a database for storing metadata corresponding to the original data, and the metadata repository is usually a relational database. The remote interaction instruction can be a control instruction for connecting the execution end of the distributed file system and the metadata repository to perform data interaction operation. The second storage path can be a path in the distributed file system for storing data describing the initial metadata.

[0077] Continuing to refer to Figure 3 The execution end of the HDFS, such as the Linux execution server 350, can be connected to the metadata repository, which can be used to store the metadata of the original data and is usually a relational database, such as an Oracle database, a mySQL database, etc. The remote interaction instruction is run on the execution end of the HDFS to obtain the initial metadata from the metadata repository based on the remote interaction instruction. For example, the execution end can obtain a string from the metadata repository through an Expect command to obtain the initial metadata.

[0078] After obtaining the initial metadata, the initial metadata can be stored in a second storage path of the HDFS distributed storage system. Through the above steps, the initial metadata corresponding to the raw data can be obtained from the relational database, which is used as the data basis for subsequent statistical analysis of business attributes. In addition, collecting raw data from different file systems can avoid the problem of missing some index information caused by using a single file.

[0079] In an example embodiment of the present disclosure, obtaining initial metadata from the metadata repository based on the remote interaction instruction includes: obtaining a pre-constructed query script, obtaining an initial metadata table from the metadata repository; based on the query script, the initial metadata table is parsed and processed to obtain a metadata result table; obtaining a data extraction component, and storing the metadata result table to a second storage path of a distributed file system as initial metadata by using the data extraction component.

[0080] The query script can be an execution script composed of Structured Query Language (SQL) statements. The initial metadata table can be a related data table for describing the initial metadata. The parsing process can be an operation process of parsing and processing the initial metadata table based on the query script. The data extraction component can be an operation component for data migration, and the data extraction component can be used to import a single table in the relational database into the distributed file system.

[0081] The specific implementation steps of the execution server 350 for obtaining the initial metadata from the metadata repository based on the remote interaction instruction are as follows: the execution server 350 can first connect to the metadata repository (such as the Hive Metadata database) 360, such as the relational database Oracle. The initial metadata table corresponding to the raw data is obtained from the metadata repository. The initial metadata table can include a main table (dbs table), a metadata table related to a view (tbls table), a metadata table storing basic information (sds table), and a partition table (partitions table).

[0082] After obtaining the above initial metadata table, the query script (such as the Oracle SQL script) can be executed by the execution server to parse and process the initial metadata table in conjunction with the SQL script to obtain the metadata result table tb_hive_metadata_result. The process of parsing and processing the initial metadata table can be based on the metadata repository. After obtaining the metadata result table, a data extraction component for data transmission between different storage systems can be obtained, and the metadata result table is stored to a second storage path of a distributed file system as initial metadata 370 by using the data extraction component.

[0083] For example, the data extraction component can be a Sqoop component, which is a data migration tool mainly used for data transfer between Hadoop (Hive) and traditional databases (such as mysql, postgresql), and can import data in a relational database (such as MySQL, Oracle, Postgres, etc.) into HDFS of Hadoop, or import data in HDFS into a relational database.

[0084] Through the data extraction component, the obtained metadata result table can be uploaded to a second storage path of HDFS, such as / user / xxx / tb_hive_metadata. Through the above steps, the metadata of the original data can be uploaded to the distributed file system as the data basis for subsequent business attribute analysis.

[0085] In an exemplary embodiment of the present disclosure, for step S120, the mirror file and the initial metadata each correspond to a mirror file metadata table and a metadata result table, including: creating a metadata table corresponding to the mirror file in the data warehouse as the mirror file metadata table; creating a metadata table corresponding to the initial metadata in the data warehouse as the metadata result table; and taking the mirror file metadata table and the metadata result table as external tables of the data file in the distributed file system.

[0086] The metadata table corresponding to the mirror file can be a result table for describing the data composition of the mirror file. The metadata table corresponding to the initial metadata can be a result table for describing the data composition of the initial metadata. The external table can refer to a data table that does not exist in the distributed file system.

[0087] After the mirror file and the initial metadata are stored in the HDFS distributed storage system, the mirror file and the initial metadata each correspond to a mirror file metadata table and a metadata result table. Referring to Figure 4 , Figure 4 A flowchart for business attribute analysis by integrating the mirror file and the initial metadata of the original data according to an exemplary embodiment of the present disclosure is schematically shown. Figure 4 The HDFS system stores HDFS files 401 such as the mirror file and the initial metadata.

[0088] For the mirror image file stored in the HDFS, a metadata table corresponding to the mirror image file can be created in the data warehouse, which can contain information about the organization, data fields and their relationships of the mirror image file as the metadata table of the mirror image file 402, and the metadata table of the mirror image file 402 can be represented as tb_hdfs_metadata. The specific content contained in the metadata table of the mirror image file 402 is shown in Table 1.

[0089] Table 1 Metadata table of mirror image file

[0090] Field name Field attribute Field description PATH STRING Path REPL INT Replica number MODIFICATION_TIME STRING Last modification time ACCESSTIME STRING Last access time PREFERREDBLOCKSIZE INT Preferred block size BLOCKCOUNT DOUBLE Total block number FILESIZE DOUBLE File size NSQUOTA INT Name quota DSQUOTA INT Monitoring quota PERMISSION STRING File permission USERNAME STRING User name GROUPNAME STRING User group name

[0091] Similarly, for the initial metadata stored in the HDFS, a metadata table corresponding to the initial metadata can be created in the data warehouse as the metadata result table 403, and the metadata result table 403 is represented as tb_hive_metadata. The specific content contained in the metadata result table 403 is shown in Table 2.

[0092] Table 2 Metadata result table

[0093]

[0094]

[0095] For the created metadata table of the mirror image file and the metadata result table, they can be used as external tables of the data files in the distributed file system. By creating the metadata table of the mirror image file and the metadata result table in the data warehouse as external tables of the HDFS, subsequent data processing can be performed using the Hive type SQL statement, which reduces the processing difficulty of data analysis for business attributes.

[0096] In an exemplary embodiment of the present disclosure, for step S130, based on the merging and parsing processing of the temporary tables corresponding to the metadata table of the mirror image file and the metadata result table, an attribute statistical result table of the business attributes corresponding to the original data is generated, including: creating the metadata table of the mirror image file and the metadata result table respectively, each corresponding to a mirror image file temporary table and a metadata temporary table; merging the mirror image file temporary table and the metadata temporary table to generate a merged data table; and parsing the merged data table to generate an attribute statistical result table.

[0097] The mirror image file temporary table can be a data table established in the system temporary folder after the mirror image file metadata table is aggregated and grouped. The metadata temporary table can be a data table established in the system temporary folder after the metadata result table is partitioned and aggregated. The merged data table can be a data table obtained by field fusion processing of the mirror image file temporary table and the metadata temporary table.

[0098] After the mirror file metadata table tb_hdfs_metadata and the metadata result table tb_hive_metadata are respectively created in the data warehouse, the mirror file temporary table and the metadata temporary table corresponding to the mirror file metadata table and the metadata result table respectively can be created according to specific data use requirements. It is continued to refer to Figure 4 The mirror file metadata table 402 is parsed and processed to obtain the mirror file temporary table 404 corresponding to the mirror file metadata table 402, and the mirror file temporary table 404 can be denoted as tmp_smallfile_1, and the specific content is shown in Table 3. The metadata result table 403 is parsed and processed to obtain the metadata temporary table 405 corresponding to the metadata result table 403, and the metadata temporary table 405 can be denoted as tmp_smallfile_2, and the specific content is shown in Table 4.

[0099] Table 3 Mirror file temporary table

[0100]

[0101]

[0102] Table 4 Metadata temporary table

[0103] Field name Field attribute Field description DB_NAME STRING Database name TBL_NAME STRING Table name TB_CREATE_TIME STRING Table creation time TBL_TYPE STRING Table type OWNER STRING Table owner OWNER_NAME STRING Database owner OWNER_TYPE STRING Database owner type LOCATION STRING Table location PAR_CREATE_TIME STRING Partition creation time PART_NAME STRING Partition name PATH STRING Partition path PAR_FLAG INT Judge whether there is partition

[0104] In an exemplary embodiment of the present disclosure, the mirror file metadata table and the metadata result table are respectively created, and the mirror file temporary table and the metadata temporary table corresponding to each other, including: performing an aggregated grouping processing on the mirror file metadata table to obtain the mirror file temporary table corresponding to the mirror file metadata table; performing a partitioned aggregation processing on the metadata result table to obtain the metadata temporary table corresponding to the metadata result table.

[0105] The aggregated grouping processing can be a processing process of integrating and counting the file information in the mirror file metadata table. The partitioned aggregation processing can be a processing process of integrating and counting the file information in different partitions of the metadata result table.

[0106] It is continued to refer to Figure 4In the process of creating the image file temporary table 404, the image file metadata table 402 can be subjected to an aggregated grouping process, such as a basic calculation and formatting process operation of summing up the number of files and the file size in the image file metadata table 402. Similarly, for the metadata result table 403, the table fields related to the data partition in the metadata result table 403 can be integrated, subjected to a partition aggregation process, and the metadata temporary table 405 corresponding to the metadata result table 403 is obtained. By establishing the temporary table corresponding to the data table, the subsequent data processing mechanism of the temporary table can fully analyze the original data and increase the reliability of the attribute indicators finally obtained.

[0107] In an exemplary embodiment of the present disclosure, the image file metadata table is subjected to an aggregated grouping process to obtain an image file temporary table corresponding to the image file metadata table, including: obtaining original table information of the image file metadata table, performing regular matching processing on the original table information to obtain valid table information of the image file metadata table; performing statistical processing on the number of files in the image file metadata table to obtain file statistical information corresponding to the image file metadata table; and generating the image file temporary table based on the valid table information and the file statistical information.

[0108] The original table information can be related information of the original file contained in the image file metadata table. The regular matching processing can be a character matching process using a regular expression on the original table information in the image file metadata table. The valid table information can be table information obtained after the regular matching processing on the original table information. The number of files can be the specific number of files contained in the image file metadata table. The file statistical information can be information obtained after statistical processing on the number of file blocks and the size of the files in the image file metadata table.

[0109] In the process of creating the image file temporary table, the original table information of the image file metadata table can be obtained, for example, the original table information can include the complete HDFS path (including the protocol number, port number, etc.) of the related table data in the image file metadata table, the data generation time, the data update time, etc. After obtaining the above original table information, the original table information can be subjected to regular matching processing to obtain valid table information of the image file metadata table.

[0110] For example, the complete HDFS path is subjected to regular matching processing to filter out the protocol number and port number in the path and obtain the valid HDFS path corresponding to the table data. For the data generation time and the data update time of the image file metadata table, the above times can also be processed by regular matching to make the time format meet the visualization requirements and obtain valid time information.

[0111] Further, the number of files in the mirror file metadata table can be statistically processed to determine the number of files (file_nums), the number of file blocks (blockcounts), the size of each file (filesize), the average file size (avg_filesize), and the like involved in the data of the mirror file metadata table, as file statistical information corresponding to the mirror file metadata table. After obtaining the above valid table information and file statistical information, the mirror file temporary table can be generated based on the valid table information and the file statistical information. Based on the temporary table creation mechanism, the technical personnel can determine the temporary table file that meets the analysis requirements according to the actual demand that changes dynamically, so as to perform different types of data analysis on the original data.

[0112] It should be noted that the partition aggregation processing of the metadata result table and the parsing processing of the merged data table are similar to the aggregation grouping processing of the mirror file metadata table. The related technical personnel can establish the corresponding metadata temporary table according to the specific requirements, and the specific processing process of establishing the metadata temporary table based on the partition aggregation processing will not be described in detail.

[0113] After obtaining the mirror file temporary table and the metadata temporary table, the mirror file temporary table tmp_smallfile_1 and the metadata temporary table tmp_smallfile_2 are merged to generate a merged data table. For example, the table fields contained in the mirror file temporary table and the metadata temporary table can be integrated to form unified table fields, and the merged data table 406 can be obtained, which can be represented as tmp_smallfile_3. The merged data table 406 can be specifically as shown in Table 5.

[0114] Table 5 Merged data table

[0115] Field name Field attribute Field description HIVE_DB_NAME STRING Database name HIVE_TABLE_NAME STRING Table name HIVE_TABLE_CREATE_NAME STRING Table creation time HIVE_TABLE_TYPE STRING Table type TABLE_LOCATION STRING HDFS table path PATH STRING Table path PAR_LAG INT Judge whether there is partition FILE_NUMS BIGINT File number HDFS_FILE_MAX_MOD_TIME STRING File last modification time HDFS_MAX_ACC_TIME STRING File last access time HDFS_BLOCKCOUNTS DOUBLE Block number HDFS_FILESIZES DOUBLE File size HDFS_AVG_FILESIZE DOUBLE Average file size HDFS_DIR_FILE STRING HDFS directory path

[0116] After obtaining the merged data table tmp_smallfile_3, the merged data table tmp_smallfile_3 can be parsed to generate an attribute statistical result table corresponding to the original data. Since the attribute statistical result table has relatively complete table fields, the business attributes can be analyzed more comprehensively based on the attribute statistical result table.

[0117] For example, merging data table 406 can use HDFS path data as a filtering condition to merge the image file temporary table tmp_smallfile_1 and the metadata temporary table tmp_smallfile_2, resulting in attribute statistics result table 407. Attribute statistics result table 407 can be represented by tb_smallfile_result, as shown in Table 6. Based on attribute statistics result table 407, the business attributes of the original data can be analyzed to determine the specific details of the business attributes corresponding to the original data.

[0118] Table 6 Attribute Statistics Results

[0119]

[0120]

[0121] In one exemplary embodiment of this disclosure, a temporary table for the image file and a temporary table for the metadata are merged to generate a merged data table. This includes: determining the first table field corresponding to the temporary table for the image file; determining the second table field corresponding to the temporary table for the metadata; and merging the temporary table for the image file and the temporary table for the metadata based on the first table field and the second table field to obtain the merged data table.

[0122] The first table field can be a table field contained in the temporary table of the image file. The second table field can be a table field contained in the temporary metadata table.

[0123] Continue to refer to Figure 4 After creating the temporary table 404 (mirror file) and the temporary table 405 (metadata), the fields corresponding to the first table in the temporary table 404 and the fields corresponding to the second table in the temporary table 405 can be determined. During table merging, the fields in the first and second tables can be compared to consolidate duplicate fields. Furthermore, new fields not included in the temporary tables 404 and 405 can be created based on attribute analysis requirements and used as fields in the merged data table. These merged fields, along with the newly added fields, form the corresponding fields in the merged data table. By integrating the data from these temporary tables, related attribute data can be merged into a single table, facilitating subsequent business attribute analysis and improving its accuracy.

[0124] In an example embodiment of the present disclosure, for step S140, the attribute statistical result table is subjected to data screening processing to obtain target attribute statistical data corresponding to the business attribute, including: obtaining a merged data table generated based on the mirror file temporary table and the metadata temporary table; performing merging processing on the merged data table and the attribute statistical result table to generate an initial statistical result table; obtaining a pre-configured attribute sorting index, and performing sorting processing on the attribute data of the initial statistical result table based on the attribute sorting index to obtain the target attribute statistical data.

[0125] The initial statistical result table can be an attribute statistical table obtained by performing field fusion processing on the merged data table and the attribute statistical result table. The attribute sorting index is a judgment index for performing sorting processing on the attributes contained in the initial statistical result table.

[0126] With reference to Figure 4 After the obtained merged data table 406 and the attribute statistical result table 407, the merged data table 406 and the attribute statistical result table 407 can be subjected to merging processing to generate an initial statistical result table. For example, the merged data table 406 and the attribute statistical result table 407 can be subjected to merging processing according to the HDFS path, table field, etc. of the two tables to generate an initial statistical result table reflecting the business attributes corresponding to the original data.

[0127] Since the initial statistical result table contains multiple business attributes, the technical personnel can configure an attribute sorting index according to the specific attribute analysis requirement. The pre-configured attribute sorting index is applied to the initial statistical result table, the attribute data of the initial statistical result table can be subjected to sorting processing based on the attribute sorting index to obtain target attribute statistical data. The technical personnel can adjust the custom sorting index at any time according to the dynamic change of the use analysis requirement in the data use process, and can sort the attribute index of the data table based on the custom sorting to achieve the business requirement of dynamically sorting and analyzing the business attribute.

[0128] The target attribute statistical data can include related attribute data sorted in the top N positions, and the target attribute statistical data can be represented as tb_smallfile_result_topN, and the specific content is shown in Table 7.

[0129] Table 7 Target attribute statistical data (Top10)

[0130] Field name Field attribute Field description DB_NAME STRING Database name TABLE_NAME STRING Table name CREATE_TIME DATE Creation time MODIFY_TIME DATE Modification time ACCESS_TIME DATE Access time TABLE_TYPE STRING table type PAR_FLAG INT whether there is partition FILE_NUMS BIGINT file number BLOCKCOUNTS DOUBLE block number FILESIZES STRING total file size AVG_FILESIZE STRING average file size DIR_COUNT BIGINT directory number TABLE_LOCATION STRING table path

[0131] In an example embodiment of the present disclosure, a pre-configured template engine is obtained; the template engine is used to extract to-be-displayed data from the target attribute statistical data; and the to-be-displayed data is sent to a visual execution end to display the to-be-displayed data through a visual interface of the visual execution end.

[0132] The template engine can be used to render and display the data to the user based on the tags. The data to be displayed can be the relevant attribute data to be visualized. The visualization execution end can be a device execution end providing a visualization interface. The visualization interface can be a device interface for displaying data.

[0133] After determining the target attribute statistical data, a pre-configured template engine can be obtained. For example, the template engine can be a Thymeleaf engine, which is a modern server-side Java template engine. The Thymeleaf engine can be integrated with web page front-end (web) frameworks such as SpringMvc and springBoot, and can render and process data through thymeleaf tags to display to the user.

[0134] In this embodiment, the Thymeleaf engine can be integrated in the springBoot framework to extract the data to be displayed from the target attribute statistical data. For example, the Thymeleaf engine can extract the top 10 attribute data from the target attribute statistical data as the data to be displayed. For example, the data to be displayed can include attribute values corresponding to attribute fields such as serial number, database name, table name, creation time, modification time, access time, table type, whether there is partition, file number, block number, file size, average file size, and directory number.

[0135] After obtaining the data to be displayed, the data to be displayed can be sent to the visualization execution end, and the data to be displayed can be displayed in the page of the visualization interface of the visualization execution end, so that the technical personnel can directly lock the business table name that needs to be rectified through the visualization interface to directly observe the specific content of different business attributes and improve the operation and maintenance efficiency.

[0136] In an exemplary embodiment of the present disclosure, a pre-configured content interception component is obtained; the page content in the visualization interface is intercepted by the content interception component to obtain alarm auxiliary data; and alarm information is generated based on the alarm auxiliary data.

[0137] The content interception component can be an operation component for intercepting specific content. The interception process can be a process of obtaining specific data content using the content interception component. The alarm auxiliary data can be related data for generating alarm information. The alarm information can be information for generating an alarm prompt.

[0138] For the data displayed in the visualization interface, the data in the visualization interface can be intercepted by a content interception component. For example, the content interception component can be selenium, an automated testing tool for web applications. Selenium testing runs directly in the browser, just like a real user is operating. The page content in the visualization interface is intercepted by the content interception component Selenium to obtain alarm auxiliary data, such as data captured by screenshot. The content interception component Selenium can capture the number of files in the displayed data in the visualization interface as alarm auxiliary data, and analyze the size of the file data in different databases.

[0139] Alarm information is generated according to the captured alarm auxiliary data, and the generated alarm information is sent. For example, the alarm information can include specific values of the business attributes corresponding to the original data, that is, the Hive table attribute statistical set of the original data. Through the above processing steps, the Hive operation and maintenance personnel can timely understand the attribute change situation, provide data support for solving the Hive performance problem, and accurately locate the problem.

[0140] It should be noted that the terms "first", "second", and the like used in the present disclosure are only used to distinguish the table fields of different data tables, and should not impose any limitation on the present disclosure.

[0141] In summary, the data processing method of the present disclosure obtains the mirror image file corresponding to the original data in the distributed file system, and obtains the initial metadata corresponding to the original data. The mirror image file metadata table and the metadata result table corresponding to the mirror image file and the initial metadata are respectively created. Based on the temporary tables corresponding to the mirror image file metadata table and the metadata result table, a merge and analysis processing is performed to generate an attribute statistical result table of the business attributes corresponding to the original data. The attribute statistical result table is subjected to data screening processing to obtain target attribute statistical data corresponding to the business attributes. On the one hand, a scheme is provided for analyzing the attribute information of the business table in the data warehouse by integrating the mirror image file and the metadata, avoiding the omission of part of the index information caused by analyzing a single file. On the other hand, by performing data merging and screening processing on the temporary tables of the mirror image file analysis table and the metadata analysis table, the original data can be fully analyzed, and the credibility of the final attribute statistical result data can be improved. On the other hand, by customizing the data screening index, the dynamic sorting function of the business attribute can be realized, avoiding the defect that only static sorting can be performed in the process of analyzing the business attribute.

[0142] It is to be understood that even though various steps of the methods of the present application are described in a particular order in the figures, this is not required or implied in any way as to the order of execution or that all illustrated steps be necessarily performed to achieve desirable results. Additional or alternative steps can be employed, steps can be omitted, a number of steps can be combined into a single step, and / or a single step can be broken into multiple steps, etc.

[0143] In addition, in the present example embodiment, a data processing apparatus is also provided. Referring to Figure 5 The data processing apparatus 500 can include a data acquisition module 510, a metadata table creation module 520, a merging processing module 530, and a data screening module 540.

[0144] Specifically, the data acquisition module 510 is configured to acquire a mirror file corresponding to the original data in the distributed file system, and acquire initial metadata corresponding to the original data; the metadata table creation module 520 is configured to create a mirror file metadata table and a metadata result table corresponding to the mirror file and the initial metadata, respectively; the merging processing module 530 is configured to perform merging and parsing processing based on temporary tables corresponding to the mirror file metadata table and the metadata result table, respectively, to generate an attribute statistical result table of a business attribute corresponding to the original data; and the data screening module 540 is configured to perform data screening processing on the attribute statistical result table to obtain target attribute statistical data corresponding to the business attribute.

[0145] In an example embodiment of the present disclosure, the data acquisition module 510 includes a mirror file acquisition unit configured to: in response to a mirror file extraction instruction, acquire a mirror file corresponding to the original data from the distributed file system; perform file parsing processing on the mirror file to obtain a parsed mirror file; and store the parsed mirror file to a first storage path of the distributed file system.

[0146] In an example embodiment of the present disclosure, the data acquisition module 510 includes a metadata acquisition unit configured to: connect an execution end of the distributed file system to a metadata repository of the original data; generate a remote interaction instruction based on the execution end, acquire initial metadata from the metadata repository based on the remote interaction instruction; and store the initial metadata to a second storage path of the distributed file system.

[0147] In an example embodiment of the present disclosure, the metadata acquisition unit includes a metadata acquisition subunit configured to: acquire a pre-constructed query script, acquire an initial metadata table from the metadata repository; perform parsing processing on the initial metadata table based on the query script to obtain a metadata result table; acquire a data extraction component, and store the metadata result table to the second storage path of the distributed file system as the initial metadata using the data extraction component.

[0148] In an example embodiment of the present disclosure, the metadata creation module 520 includes a metadata creation unit configured to: create a metadata table corresponding to the mirror file in the data warehouse as a mirror file metadata table; create a metadata table corresponding to the initial metadata in the data warehouse as a metadata result table; and take the mirror file metadata table and the metadata result table as external tables of the data file in the distributed file system.

[0149] In an example embodiment of the present disclosure, the merging processing module 530 includes a merging processing unit configured to: create a mirror file temporary table and a metadata temporary table respectively corresponding to the mirror file metadata table and the metadata result table; perform merging processing on the mirror file temporary table and the metadata temporary table to generate a merged data table; and perform parsing processing on the merged data table to generate an attribute statistical result table.

[0150] In an example embodiment of the present disclosure, the merging processing unit includes a temporary table creation unit configured to: perform aggregate grouping processing on the mirror file metadata table to obtain a mirror file temporary table corresponding to the mirror file metadata table; and perform partition aggregate processing on the metadata result table to obtain a metadata temporary table corresponding to the metadata result table.

[0151] In an example embodiment of the present disclosure, the temporary table creation unit includes a temporary table creation subunit configured to: obtain original table information of the mirror file metadata table, perform regular matching processing on the original table information to obtain valid table information of the mirror file metadata table; perform statistical processing on the number of files in the mirror file metadata table to obtain file statistical information corresponding to the mirror file metadata table; and generate the mirror file temporary table based on the valid table information and the file statistical information.

[0152] In an example embodiment of the present disclosure, the merging processing unit includes a merging processing subunit configured to: determine a first table field corresponding to the mirror file temporary table; determine a second table field corresponding to the metadata temporary table; and perform merging processing on the mirror file temporary table and the metadata temporary table based on the first table field and the second table field to obtain the merged data table.

[0153] In an example embodiment of the present disclosure, the data filtering module 540 includes a data filtering unit configured to: obtain the merged data table generated based on the mirror file temporary table and the metadata temporary table; perform merging processing on the merged data table and the attribute statistical result table to generate an initial statistical result table; obtain a pre-configured attribute sorting index; perform sorting processing on attribute data of the initial statistical result table based on the attribute sorting index to obtain target attribute statistical data.

[0154] In an example embodiment of the present disclosure, the data processing apparatus 500 further comprises a data display module, configured to: acquire a pre-configured template engine; extract the data to be displayed from the target attribute statistical data by using the template engine; and send the data to be displayed to a visualization execution end, and display the data to be displayed through a visualization interface of the visualization execution end.

[0155] In an example embodiment of the present disclosure, the data processing apparatus 500 further comprises an alarm information generation module, configured to: acquire a pre-configured content interception component; perform interception processing on page content in the visualization interface by using the content interception component to obtain alarm auxiliary data; and generate alarm information based on the alarm auxiliary data.

[0156] The specific details of the virtual modules of the data processing apparatuses in the above embodiments have been described in detail in the corresponding data processing methods, and thus will not be described here again.

[0157] It should be noted that, although several modules or units of the data processing apparatus are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for embodiment.

[0158] In addition, in the example embodiments of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0159] Those skilled in the art can understand that each aspect of the present disclosure can be implemented as a system, a method or a program product. Therefore, each aspect of the present disclosure can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0160] The electronic device 600 according to this embodiment of the present disclosure will be described below with reference to Figure 6 . Figure 6 The electronic device 600 shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0161] As Figure 6 shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to, the above-mentioned at least one processing unit 610, the above-mentioned at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0162] The storage unit stores program codes which can be executed by the processing unit 610, so that the processing unit 610 performs the steps described in the above "Exemplary Methods" section according to various exemplary embodiments of the present disclosure.

[0163] The storage unit 620 can include a readable medium in the form of volatile storage such as a random access memory (RAM) 621 and / or cache memory 622, and also can further include a read-only memory (ROM) 623.

[0164] The storage unit 620 can include program / utility 624 having a set of programs / modules 625, including but not limited to, operating systems, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment, alone or in combination.

[0165] The bus 630 can represent one or more of several types of bus structures, including a storage bus or bus controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures.

[0166] The electronic device 600 can also communicate with one or more external devices 670 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices such as a storage device or an external effects device; and / or one or more devices that enable a user to interact with the electronic device 600; and / or one or more devices that enable the electronic device 600 to communicate with one or more other computing devices. Such communication can be facilitated by an Input / Output (I / O) interface 650. Still yet, the electronic device 600 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet, via a network adapter 660. As depicted, the network adapter 660 is in communication with the other components of the electronic device 600 through the bus 630. It should be appreciated that although not shown, other hardware and / or software modules could be used in connection with the electronic device 600. Such modules include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0167] Those skilled in the art can easily understand from the above description of the embodiments that the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0168] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of the present disclosure is stored. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the above-mentioned “example method” section according to various example embodiments of the present disclosure when the program product is run on the terminal device.

[0169] Reference Figure 7 As shown, a program product 700 for implementing the above-mentioned method according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or apparatus.

[0170] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0171] A computer readable signal medium can include a propagated data signal with computer executable code embodied therein. For example, a propagated signal can be an electromagnetic signal, an optical signal, and / or any other suitable type of signal. Such a propagated signal can be in the form of electrical magnetic waves, optical waves, and / or any other suitable type of waves upon which computer executable code is embodied. A suitable medium for storing and / or transmitting computer readable code includes one or more types of random access memory (RAM), magnetic storage, optical storage, and / or any other suitable type of storage.

[0172] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0173] Computer readable program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider. The application program code can be downloaded to the user's computing device from an external computing device or server through any type of network, including a local area network, a wide area network, or the Internet using a browser or other applet.

[0174] Furthermore, the above-described figures are merely schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is readily understood that the processes shown in the above-described figures do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0175] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practice of the present disclosure. The present application is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including modifications and equivalents of the present disclosure that are obvious to those skilled in the art. The specification and examples are to be regarded as illustrative only and the true scope and spirit of the present disclosure is indicated by the appended claims.

[0176] It should be understood that the present disclosure is not limited to the precise construction that has been described and illustrated above and that various modifications and changes can be made without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims appended hereto.

Claims

1. A data processing method, characterized in that, include: Obtain the image file corresponding to the original data in the distributed file system, and obtain the initial metadata corresponding to the original data; Creating a metadata table and a metadata result table corresponding to the image file and the initial metadata respectively, including: creating a metadata table corresponding to the image file in the data warehouse as the image file metadata table; and creating a metadata table corresponding to the initial metadata in the data warehouse as the metadata result table. Based on the temporary tables corresponding to the image file metadata table and metadata result table, a merge and parsing process is performed to generate an attribute statistics result table for the business attributes corresponding to the original data. The attribute statistics result table is processed by data filtering to obtain the target attribute statistics data corresponding to the business attribute.

2. The method according to claim 1, characterized in that, The step of obtaining the image file corresponding to the original data in the distributed file system includes: In response to the image file extraction command, the image file corresponding to the original data is obtained from the distributed file system; The image file is parsed to obtain a parsed image file; The parsed image file is stored in the first storage path of the distributed file system.

3. The method according to claim 1, characterized in that, The step of obtaining the initial metadata corresponding to the original data includes: Connect the execution end of the distributed file system to the metadata repository of the original data; Based on the execution terminal, a remote interaction instruction is generated, and based on the remote interaction instruction, the initial metadata is obtained from the metadata repository; The initial metadata is stored in the second storage path of the distributed file system.

4. The method according to claim 3, characterized in that, The step of retrieving the initial metadata from the metadata repository based on the remote interaction command includes: Obtain a pre-built query script and retrieve the initial metadata table from the metadata repository; The initial metadata table is parsed based on the query script to obtain a metadata result table; Obtain the data extraction component, and use the data extraction component to store the metadata result table to the second storage path of the distributed file system as the initial metadata.

5. The method according to claim 1, characterized in that, The step of creating image file metadata tables and metadata result tables corresponding to the image file and the initial metadata respectively also includes: The image file metadata table and the metadata result table are used as external tables for data files in the distributed file system.

6. The method according to claim 1, characterized in that, The process of merging and parsing the temporary tables corresponding to the metadata table and metadata result table of the image file to generate the attribute statistics result table of the business attributes corresponding to the original data includes: Create the image file metadata table and metadata result table, as well as their respective temporary image file tables and temporary metadata tables; The temporary table of the image file and the temporary table of metadata are merged to generate a merged data table; The merged data table is parsed to generate the attribute statistics result table.

7. The method according to claim 6, characterized in that, The creation of the image file metadata table and metadata result table, and their corresponding image file temporary table and metadata temporary table, includes: The image file metadata table is aggregated and grouped to obtain a temporary image file table corresponding to the image file metadata table; The metadata result table is partitioned and aggregated to obtain a temporary metadata table corresponding to the metadata result table.

8. The method according to claim 7, characterized in that, The aggregation and grouping process of the image file metadata table to obtain the image file temporary table corresponding to the image file metadata table includes: Obtain the original table information of the image file metadata table, perform regular expression matching on the original table information, and obtain the valid table information of the image file metadata table; The number of files in the image file metadata table is statistically processed to obtain the file statistics information corresponding to the image file metadata table; Based on the valid table information and the file statistics information, the temporary table for the mirror file is generated.

9. The method according to claim 6, characterized in that, The process of merging the temporary table of the image file and the temporary table of metadata to generate a merged data table includes: Determine the first table field corresponding to the temporary table of the image file; Determine the fields of the second table corresponding to the temporary metadata table; Based on the fields of the first table and the second table, the temporary table of the image file and the temporary table of metadata are merged to obtain the merged data table.

10. The method according to claim 1, characterized in that, The step of filtering the attribute statistics result table to obtain the target attribute statistics data corresponding to the business attribute includes: Obtain the merged data table generated based on the temporary table of the image file and the temporary table of metadata; The merged data table and the attribute statistics result table are merged to generate an initial statistics result table; Obtain pre-configured attribute ranking indicators, and sort the attribute data of the initial statistical result table based on the attribute ranking indicators to obtain the target attribute statistical data.

11. The method according to claim 1, characterized in that, The method further includes: Get a pre-configured template engine; The template engine is used to extract the data to be displayed from the target attribute statistics. The data to be displayed is sent to the visualization execution terminal, and the data to be displayed is displayed through the visualization interface of the visualization execution terminal.

12. The method according to claim 11, characterized in that, The method further includes: Retrieve pre-configured content capture components; The content capture component is used to capture and process the page content in the visualization interface to obtain alarm auxiliary data. Alarm information is generated based on the alarm auxiliary data.

13. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire the image file corresponding to the original data in the distributed file system, and to acquire the initial metadata corresponding to the original data. The metadata table creation module is used to create a metadata table and a metadata result table corresponding to the image file and the initial metadata, respectively, including: creating a metadata table corresponding to the image file in the data warehouse as the image file metadata table; and creating a metadata table corresponding to the initial metadata in the data warehouse as the metadata result table. The merging processing module is used to perform merging and parsing processing based on the temporary tables corresponding to the metadata table and metadata result table of the image file, and generate the attribute statistics result table of the business attributes corresponding to the original data. The data filtering module is used to filter the data in the attribute statistics result table to obtain the target attribute statistics data corresponding to the business attribute.

14. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the data processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the data processing method according to any one of claims 1 to 12.

Images