Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and a computer readable storage medium. The data processing method comprises the following steps: based on the information of a data source, registering a source table from the data source as a persistent Hive external table; obtaining index data from the persistent Hive external table through Spark SQL; importing the obtained index data into a result table; calling the index data from the result table, and writing the called index data into a pre-created database; and exporting the data in the database for display. The method of the application registers the source table from the data source as a persistent Hive external table, and since the persistent Hive external table can be repeatedly used, the operation of editing the configuration items of the data source and the table and creating a view is avoided, the data processing process is simplified, the data processing efficiency can be effectively improved, the data calculation is easy to use, the scalability is good, and the needs of practical applications can be well met.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In recent years, big data technology has developed rapidly, and various big data systems have entered people's daily lives, such as government service big data systems. Big data systems require the aggregation and analysis of data from various business systems to ultimately generate indicator results with business attributes, which are then displayed on a screen in real time. For example, in the rapidly developing government service big data systems, the leadership dashboard module is used to aggregate and analyze data from various business systems, ultimately generating indicator results with business attributes and displaying them on a screen in real time. In traditional big data service systems, various business systems are scattered, and data storage is inconsistent. Each time data from a data source is processed, it is necessary to edit the configuration items of the data source and table separately and create views, which leads to cumbersome data processing workflows and low data processing efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method, apparatus, electronic device, and computer-readable storage medium. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0004] According to one aspect of the embodiments of this application, a data processing method is provided, applied to a computing device, the method comprising:

[0005] Based on the information from the data source, the source table from the data source is registered as a persistent Hive external table;

[0006] Retrieve metric data from the persistent Hive external table using Spark SQL;

[0007] Import the acquired indicator data into the results table;

[0008] Retrieve the indicator data from the result table and write the retrieved indicator data into a pre-created database;

[0009] Export the data from the database for display.

[0010] In some embodiments of this application, registering the source table from the data source as a persistent Hive external table based on the information from the data source includes:

[0011] Extract the fixed configuration information corresponding to the source table from the information of the data source;

[0012] The source table and the fixed configuration information are assembled to generate a table creation statement;

[0013] The table creation statement is executed using Spark SQL to generate the persistent Hive external table.

[0014] In some embodiments of this application, obtaining metric data from the persistent Hive external table via Spark SQL includes:

[0015] The Spark SQL query retrieves all initial data for the metrics from the persistent Hive external table.

[0016] The indicator data is obtained by summarizing and calculating all the initial indicator data using Spark SQL.

[0017] In some embodiments of this application, writing the retrieved indicator data into a pre-created database includes:

[0018] Determine whether a preset table exists in the pre-created database that matches the retrieved indicator data;

[0019] If the preset table does not exist, a new table matching the retrieved indicator data is created, and the indicator data is imported into the newly created matching table.

[0020] If the preset table exists, the preset table is compared with the corresponding information of the indicator data to obtain the comparison result;

[0021] Based on the comparison results, the preset table is updated using the retrieved indicator data, employing an update method corresponding to the comparison results.

[0022] In some embodiments of this application, updating the preset table using the retrieved indicator data according to the comparison result and employing an update method corresponding to the comparison result includes:

[0023] If the comparison result shows that the corresponding information of the preset table and the indicator data match, then the retrieved indicator data is added to the preset table, or the preset table is cleared and the retrieved indicator data is imported into the cleared preset table.

[0024] If the comparison result shows that the corresponding information in the preset table and the indicator data does not match, then the mismatched fields in the preset table are updated using the retrieved indicator data.

[0025] In some embodiments of this application, updating mismatched fields in the preset table using the retrieved indicator data includes:

[0026] For fields in the preset table that do not match the retrieved indicator data, the mismatched fields are replaced with the corresponding fields from the retrieved indicator data.

[0027] In some embodiments of this application, updating mismatched fields in the preset table using the retrieved indicator data includes:

[0028] If a field is missing in the corresponding position in the preset table and in the retrieved indicator data, then the field in the retrieved indicator data at the corresponding position is inserted into the corresponding position in the preset table.

[0029] According to another aspect of the embodiments of this application, a data processing apparatus is provided, applied to a computing device, the apparatus comprising:

[0030] The registration module is used to register source tables from the data source as persistent Hive external tables based on information from the data source.

[0031] The acquisition module is used to retrieve metric data from the persistent Hive external table using Spark SQL.

[0032] The import module is used to import the acquired indicator data into the result table;

[0033] The writing module is used to retrieve the indicator data from the result table and write the retrieved indicator data into a pre-created database.

[0034] The export module is used to export data from the database for display.

[0035] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the data processing methods described above.

[0036] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which is executed by a processor to implement the data processing method of any of the above.

[0037] One aspect of the technical solution provided in this application embodiment may include the following beneficial effects:

[0038] The data processing method provided in this application embodiment can register source tables from multiple data sources as a persistent Hive external table, import the metric data from the persistent Hive external table into a result table using Spark SQL, write the metric data from the result table into multiple databases, and export the data from the databases. Since the persistent Hive external table can be reused multiple times, it avoids the need to edit the configuration items of the data source and table and create views, simplifying the data processing flow. It can effectively improve data processing efficiency, enhance the usability of data computation, and produce data processing results with small errors and good scalability, thus well meeting the needs of practical applications.

[0039] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing embodiments of this application. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 The diagram illustrates an implementation environment for a data processing method provided in one embodiment of this application.

[0042] Figure 2 A flowchart of a data processing method provided in one embodiment of this application is shown;

[0043] Figure 3 It shows Figure 2 Flowchart of step S10;

[0044] Figure 4 It shows Figure 2 Flowchart of step S20;

[0045] Figure 5 It shows Figure 2 Flowchart of step S40;

[0046] Figure 6 A flowchart of a data processing method provided in another embodiment of this application is shown;

[0047] Figure 7 A block diagram of a data processing apparatus provided in one embodiment of this application is shown;

[0048] Figure 8This illustration shows a block diagram of an electronic device structure provided in one embodiment of the present application;

[0049] Figure 9 A schematic diagram of a computer-readable storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0052] Online Analytical Processing (OLAP) is a software technology that enables analysts to quickly, consistently, and interactively view information from various perspectives to achieve a deeper understanding of data. It features Fast Analysis of Shared Multidimensional Information (FASMI). Here, F stands for Fast, meaning the system can respond to most user analysis requests within seconds; A stands for Analysis, meaning users can define new specialized calculations without programming, incorporate them as part of the analysis, and receive reports in the desired manner; M stands for Multidimensional, meaning it provides multidimensional views and analyses of the data; and I stands for Information, meaning it can obtain information promptly and manage large volumes of information.

[0053] The online analytical processing engine used in this embodiment can be Spark SQL.

[0054] Figure 1This diagram illustrates an implementation environment for a data processing method provided in one embodiment of this application. The implementation environment includes a computer device and a display device. The computer device includes a processor and a memory interconnected. A display application is installed on the display device. The display device can be, for example, a display screen. The computer device and the display device can be connected via Bluetooth, USB (Universal Serial Bus), or other communication methods. The memory of the computer device may store computer-readable instructions. When these instructions are executed by the processor, the processor can execute the data processing method of this embodiment. The computer device can send the data obtained from executing the data processing method (e.g., in the form of a derived table) to the display device, and the display application can display the data from the computer device on the display device. For example, in a government service big data system, the display device can be a large display screen set up in a public place for public display. The computer device sends data to the large display screen, and the large display screen displays the data. Alternatively, the display device can be a smartphone or personal computer, where the computer device sends data to the display device, and the display device displays the data.

[0055] like Figure 2 As shown, one embodiment of this application provides a data processing method applied to a computing device, the method including steps S10 to S50. The computing device may be, for example, a computer device.

[0056] S10. Based on the information from the data source, register the source table from the data source as a persistent Hive external table.

[0057] Data sources can include databases such as MySQL, PostgreSQL, Oracle, Greenplum, HBase, Elasticsearch (ES), DM, and Vertica. This persistent Hive external table can be reused multiple times in Spark SQL. Because the persistent Hive external table can be reused, it avoids the drawback of needing to edit the configuration items of the data source and table and create views separately each time data from a data source is processed. It also avoids the need to call methods like `createGlobalTempView()` to create a global temporary view every time non-Hive tables are queried or computed using Spark SQL, thus greatly simplifying the data processing flow and improving data processing efficiency. From a business perspective, if a new data source type is added, the tables in the new data source can be registered as Hive external tables based on the characteristics of the new data source, thus improving the scalability of metric calculations. Furthermore, from a performance perspective, the Spark cluster itself is also scalable.

[0058] Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It's a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. Hive data warehouse tools can map structured data files to a database table and provide SQL query functionality, transforming SQL statements into MapReduce tasks for execution.

[0059] Hive is a data warehouse analysis system built on Hadoop. It provides rich SQL query methods to analyze data stored in the Hadoop Distributed File System: it can map structured data files to a database table and provide complete SQL query functions; it can convert SQL statements into MapReduce tasks for execution, and analyze the required content through its own SQL queries. This set of SQL is called Hive SQL, which makes it easy for users who are not familiar with MapReduce to query, summarize and analyze data using the SQL language.

[0060] Hive external tables do not load data into Hive's default repository (mounted data), reducing data transfer while allowing data sharing with other external tables. Using Hive external tables means Hive does not modify the source data, eliminating concerns about data corruption or loss.

[0061] like Figure 3 As shown, in some embodiments, step S10 includes steps S101 to S103, as follows:

[0062] S101. Extract the fixed configuration information corresponding to the source table from the information of the data source.

[0063] Based on the type of data source, you can obtain the corresponding data source information, such as connection information, port (if any), Kerberos authentication information (if any), etc.

[0064] Taking Elasticsearch data source as an example, the fixed configuration information corresponding to the aforementioned source table can be extracted from the Elasticsearch data source information, including: the Database to be used in the Hive table, table name, resource (ES source information), nodes (ES nodes), and port (port). If Kerberos authentication is enabled on the cluster, Kerberos authentication information (such as principal) must be provided.

[0065] S102. Assemble the source table and the fixed configuration information to generate a table creation statement.

[0066] The table creation statement can be an SQL table creation statement, which is used to generate a table. The aforementioned source table and the aforementioned fixed configuration information are used as the basic data for table creation.

[0067] Specifically, the connection information of the source table, table field names, field types, and query program entry points are concatenated in the SQL table creation statement to obtain a persistent Hive external table.

[0068] In some implementations, tables can be created using Spark Thrift Server. Spark Thrift Server is a Thrift service implemented by the Spark community based on HiveServer2. It executes user-sent SQL queries and retrieves results through its own programmatic calls. Apache Spark is a memory-based distributed computing engine designed for large-scale data processing. Its key feature is storing computational data and intermediate results in memory, significantly reducing I / O overhead. Leveraging the Spark computing engine and the Hadoop ecosystem, data analysis in the aforementioned government service scenarios can be effectively handled, meeting the needs of most business users.

[0069] S103. Execute the table creation statement using SparkSQL to generate the persistent Hive external table.

[0070] In this embodiment, the table creation statement can be an SQL table creation statement. Spark SQL, based on Spark Core, can uniformly process relational tables and RDDs using SQL commands, making it easy to use. A persistent Hive external table is a table that can be used directly in Spark SQL (using Spark Thrift Server and SparkSession) without needing to be created each time using the `createTempView()` or `createGlobalTempView()` methods. In big data services, such as government services, the specific table format is one source table corresponding to one Hive external table, with identical field names and types mapped to Hive types. In some implementations, Spark On Hive can be used to generate a custom external table in Hive.

[0071] S20. Obtain metric data from the persistent Hive external table using Spark SQL.

[0072] like Figure 4 As shown, in some embodiments, step S20 includes steps S201 to S202, as follows:

[0073] S201. Use Spark SQL to query all the initial data of the metrics from the persistent Hive external table.

[0074] Spark SQL provides SQL query capabilities for big data. Spark SQL is a module in Spark used to process structured data. When querying data from tables of other data sources using Spark SQL, the corresponding program can be found using information provided in the registry. The query can then be sent to other data sources and the returned results can be retrieved.

[0075] S202. The initial data of all the indicators are aggregated and calculated using Spark SQL to obtain the indicator data.

[0076] The initial data of all the indicators are aggregated and calculated using Spark SQL, including operations such as summing the initial data of all the indicators using Spark SQL, thereby integrating the initial data of each indicator to obtain the final indicator data.

[0077] S30. Import the obtained indicator data into the result table.

[0078] Each result table has a globally unique name. The result table is a Hive table. The metric data obtained in step S20 is imported into this Hive table. The result table can be viewed directly to check the accuracy of the data.

[0079] S40. Retrieve the indicator data from the result table and write the retrieved indicator data into a pre-created database.

[0080] The data in the persistent Hive external tables, after undergoing various processing steps, is ultimately displayed on a display device. For example, in a government service big data system, the final data will be displayed on a publicly accessible large display screen, which is then sent to the display application associated with that screen. To ensure accurate display of the final data, a database matching the display application needs to be pre-created. This pre-created database stores the retrieved indicator data. Various permissions can be set for this database according to the needs of the actual application. This pre-created database can be a variety of relational databases such as MySQL, PostgreSQL, Greenplum, DM, and HDFS.

[0081] like Figure 5 As shown, in some embodiments, step S40, writing the retrieved indicator data into a pre-created database, includes steps S401 to S404:

[0082] S401. Determine whether a preset table exists in the pre-created database that matches the retrieved indicator data.

[0083] For example, the retrieved indicator data is in DataFrame format, containing rows with a schema, which describes the data structure. Based on the schema, a pre-created database is searched to see if a pre-defined table matching the DataFrame's structure exists.

[0084] S402. If the preset table does not exist, a new table matching the retrieved indicator data is created, and the indicator data is imported into the newly created matching table.

[0085] For example, if a pre-created database does not have a pre-defined table that matches the structure of the DataFrame, a new table matching the DataFrame's structure is created based on the schema, and the retrieved DataFrame-formatted index data is imported into the new table. This data writing method can be called the new table writing method.

[0086] S403. If the preset table exists, the preset table is compared with the corresponding information of the indicator data to obtain the comparison result.

[0087] The corresponding information can be field names and / or data types. For example, if a pre-created database contains a preset table whose structure matches the retrieved indicator data, the preset table is compared with the corresponding information in the retrieved indicator data to obtain a comparison result. This comparison result includes whether the corresponding information in the preset table matches or does not match the corresponding information in the retrieved indicator data. A matching comparison result means that the corresponding information in the preset table is completely consistent with the corresponding information in the retrieved indicator data; a non-matching comparison result means that there are inconsistent parts between the corresponding information in the preset table and the retrieved indicator data.

[0088] In some implementations, the corresponding information between the preset table and the indicator data can be compared based on the uniqueness constraints in the preset table. For example, the uniqueness constraints can be PRIMARY KEY or UNIQUE KEY.

[0089] S404. Based on the comparison result, update the preset table using the retrieved index data by adopting an update method corresponding to the comparison result.

[0090] In some implementations, step S404 includes:

[0091] S4041. If the comparison result shows that the corresponding information of the preset table and the indicator data match, then the retrieved indicator data is added to the preset table, or the preset table is cleared and the retrieved indicator data is imported into the cleared preset table.

[0092] The data writing method of adding the retrieved indicator data to the preset table can be called the append writing method; the data writing method of clearing the preset table and importing the retrieved indicator data into the cleared preset table can be called the overwrite writing method.

[0093] S4042. If the comparison result shows that the corresponding information in the preset table and the indicator data do not match, then the mismatched fields in the preset table are updated using the retrieved indicator data.

[0094] The comparison result indicates that the corresponding information in the preset table does not match the corresponding information in the indicator data, meaning that there are parts in the preset table that are inconsistent with the corresponding information in the indicator data.

[0095] The method of updating mismatched fields in the preset table using the retrieved indicator data can be called an update write method, which may include:

[0096] For fields in the preset table that do not match the retrieved indicator data, replace the mismatched fields with the corresponding fields from the retrieved indicator data; or,

[0097] If a field is missing in the corresponding position in the preset table and in the retrieved indicator data, then the field in the retrieved indicator data at the corresponding position is inserted into the corresponding position in the preset table.

[0098] The update write method can achieve the effect of periodically updating part of the data in the database.

[0099] For example, for parts in the preset table that are inconsistent with the corresponding information in the retrieved indicator data, the corresponding parts in the preset table are replaced with the corresponding information in the retrieved indicator data; if there is a missing field in the preset table at the position corresponding to the corresponding information in the retrieved indicator data, the corresponding information in the retrieved indicator data is directly inserted into the corresponding position in the preset table.

[0100] In some implementations, the specified update condition is called the update condition, and the column to be updated is called the update column. Taking the retrieved indicator data as a DataFrame as an example, the update write process includes: comparing the update condition in the DataFrame with the update condition in a preset table; if the data for the update condition already exists in the preset table, then the data in the update column of the DataFrame is updated in the preset table; otherwise, if the data for the update condition does not exist in the preset table, then the data in the update column of the DataFrame is inserted into the preset table.

[0101] S50. Export the data from the database for display.

[0102] The data in the database is exported to a table, resulting in an exported table. This exported table can be accessed and displayed by a display application, or it can be sent to a display device for display. The display device has a display application installed that is compatible with the database, so the display application can directly access the exported table and display it on the display device.

[0103] In some implementations, steps S10-S50 can be repeated to achieve continuous data updates. The specific update time can be set according to actual needs, and can be set to a periodic update mode or a non-periodic update mode.

[0104] like Figure 6 As shown, another embodiment of the data processing method includes: registering a source table from the data source as a persistent Hive external table based on information from the data source; obtaining metric data from the persistent Hive external table via Spark SQL; importing the obtained metric data into a result table; retrieving the metric data from the result table and writing the retrieved metric data into a pre-created database; and exporting the data from the database in the form of a table, i.e., a data table, which is used for display.

[0105] This application provides a data processing method that can register source tables from multiple data sources as a persistent Hive external table, import metric data from the persistent Hive external table into a result table using Spark SQL, write the metric data from the result table into multiple databases, and export the data from the databases for display. Since the persistent Hive external table can be reused multiple times, it avoids the need to edit the configuration items of the data source and table and create views each time, simplifying the data processing flow. It can effectively improve data processing efficiency, enhance the ease of use of data computation, and produce data processing results with small errors and good scalability, thus well meeting the needs of practical applications.

[0106] like Figure 7 As shown, another embodiment of this application provides a data processing apparatus applied to a computing device, the apparatus comprising:

[0107] The registration module is used to register source tables from the data source as persistent Hive external tables based on information from the data source.

[0108] The acquisition module is used to retrieve metric data from the persistent Hive external table using Spark SQL.

[0109] The import module is used to import the acquired indicator data into the result table;

[0110] The writing module is used to retrieve the indicator data from the result table and write the retrieved indicator data into a pre-created database.

[0111] The export module is used to export data from the database for display.

[0112] In some implementations, the registration module includes:

[0113] An extraction unit is used to extract fixed configuration information corresponding to the source table from the information of the data source;

[0114] An assembly unit is used to assemble the source table and the fixed configuration information to generate a table creation statement.

[0115] The generation unit is used to execute the table creation statement through the Spark SQL to generate the persistent Hive external table.

[0116] In some implementations, the acquisition module includes:

[0117] The query unit is used to query all initial data of metrics from the persistent Hive external table using Spark SQL.

[0118] The calculation unit is used to perform summary calculations on all the initial indicator data using the Spark SQL to obtain the indicator data.

[0119] In some implementations, the writing module includes:

[0120] The judgment unit is used to determine whether there is a preset table in the pre-created database that matches the retrieved indicator data;

[0121] A new unit is created to create a new table that matches the retrieved indicator data if the preset table does not exist, and to import the indicator data into the newly created matching table.

[0122] The comparison unit is used to compare the preset table with the corresponding information of the indicator data if the preset table exists, and to obtain the comparison result.

[0123] An update unit is used to update the preset table using the retrieved indicator data, based on the comparison results and employing an update method corresponding to the comparison results.

[0124] In some implementations, the above-mentioned update unit is specifically used for:

[0125] If the comparison result shows that the corresponding information of the preset table and the indicator data match, then the retrieved indicator data is added to the preset table, or the preset table is cleared and the retrieved indicator data is imported into the cleared preset table.

[0126] If the comparison result shows that the corresponding information in the preset table and the indicator data does not match, then the mismatched fields in the preset table are updated using the retrieved indicator data.

[0127] In some implementations, updating mismatched fields in the preset table using the retrieved indicator data may include:

[0128] For fields in the preset table that do not match the retrieved indicator data, the mismatched fields are replaced with the corresponding fields from the retrieved indicator data.

[0129] In some implementations, updating mismatched fields in the preset table using the retrieved indicator data may include:

[0130] If a field is missing in the corresponding position in the preset table and in the retrieved indicator data, then the field in the retrieved indicator data at the corresponding position is inserted into the corresponding position in the preset table.

[0131] The data processing apparatus and the data processing method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0132] Another embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data processing method of any of the above embodiments.

[0133] like Figure 8As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102, and a communication interface 103. The processor 100, the communication interface 103, and the memory 101 are connected via the bus 102. The memory 101 stores a computer program that can run on the processor 100. When the processor 100 runs the computer program, it executes the method provided in any of the foregoing embodiments of this application.

[0134] The memory 101 may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0135] Bus 102 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used to store programs. After receiving an execution instruction, the processor 100 executes the program. The method disclosed in any of the foregoing embodiments of this application can be applied to the processor 100, or implemented by the processor 100.

[0136] Processor 100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 100 or by instructions in software form. The processor 100 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), an Off-the-shelf Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 101. The processor 100 reads the information in memory 101 and, in conjunction with its hardware, completes the steps of the above method.

[0137] The electronic devices and methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0138] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the data processing method of any of the above embodiments.

[0139] This application also provides a computer-readable storage medium corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 9 The computer-readable storage medium shown is an optical disc 20, on which a computer program (i.e., a program product) is stored, which, when run by a processor, executes the methods provided in any of the foregoing embodiments.

[0140] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0141] The computer-readable storage medium provided in the above embodiments of this application and the method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0142] It should be noted that:

[0143] The term "module" is not intended to be limited to a specific physical form. Depending on the application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. Furthermore, different modules may share common components or even be implemented using the same components. Clear boundaries may or may not exist between different modules.

[0144] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used with the examples based on this. The required structure for constructing such devices is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0145] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0146] The embodiments described above merely illustrate the implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method applied to a computing device, characterized in that, The method comprises: registering a source table from a data source as a persistent Hive external table based on information of the data source; the persistent Hive external table can be reused; if a new data source type is added, registering a table in the new data source as a Hive external table according to characteristics of the new data source; obtaining index data from the persistent Hive external table through a Spark SQL; importing the obtained index data into a result table; calling the index data from the result table and writing the called index data into a pre-created database; exporting data in the database for display. The method comprises: extracting fixed configuration information corresponding to the source table from the information of the data source; assembling the source table and the fixed configuration information to generate a table creation statement; executing the table creation statement through the Spark SQL to generate the persistent Hive external table.

2. The data processing method according to claim 1, characterized in that, The method comprises: querying all initial index data from the persistent Hive external table through the Spark SQL; performing summary calculation on the all initial index data through the Spark SQL to obtain the index data.

3. The data processing method of claim 1, wherein, The method comprises: judging whether a preset table matching the called index data exists in the pre-created database; if the preset table does not exist, creating a table matching the called index data and importing the index data into the created matching table; if the preset table exists, comparing corresponding information of the preset table and the index data to obtain a comparison result; according to the comparison result, updating the preset table with the called index data by using an updating mode corresponding to the comparison result.

4. The data processing method according to claim 3, characterized in that, The method comprises: if the comparison result is that the corresponding information of the preset table and the index data matches, adding the called index data into the preset table or emptying the preset table and importing the called index data into the emptied preset table; if the comparison result is that the corresponding information of the preset table and the index data does not match, updating a field in the preset table that does not match with the called index data.

5. The data processing method according to claim 4, characterized in that, The method comprises: for a field in the preset table that does not match with the called index data, replacing the field that does not match with a corresponding field in the called index data.

6. The data processing method according to claim 4, characterized in that, The method comprises: If a field is absent at the corresponding position in the preset table, a field at the corresponding position in the retrieved index data is inserted into the corresponding position in the preset table.

7. A data processing apparatus for use in a computing device, the apparatus comprising: The device comprises: a registration module, configured to register a source table from a data source as a persistent Hive external table based on information of the data source; the persistent Hive external table can be reused; if a new data source type is added, a table in the new data source is registered as a Hive external table according to characteristics of the new data source; an acquisition module, configured to acquire index data from the persistent Hive external table through a Spark SQL; an import module, configured to import the acquired index data into a result table; a write module, configured to retrieve the index data from the result table and write the retrieved index data into a pre-created database; an export module, configured to export data in the database for display; the registration module comprises: an extraction unit, configured to extract fixed configuration information corresponding to the source table from the information of the data source; an assembly unit, configured to assemble the source table and the fixed configuration information to generate a table creation statement; a generation unit, configured to execute the table creation statement through the Spark SQL to generate the persistent Hive external table.

8. An electronic device, comprising: A computer program is stored in the memory and executable on the processor, and the processor executes the program to implement the data processing method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the data processing method of any one of claims 1-6.

Images