Data processing method and device and computer readable storage medium

Abstract

The invention discloses a data processing method and device and a computer readable storage medium, and the method comprises the steps: determining a standard file based on HIVE original table data and an sftp file; polling the standard file to determine whether the CTL file is ready or not; if the CTL file is ready, analyzing the DAT file based on the XML file to obtain a target analysis file; and concurrently writing the target analysis files into an ES partition in batches. According to the method, the DAT file in the standard file is analyzed to obtain the target analysis file, the target analysis file is concurrently written into the ES partition in batches, the HIVE original table data of different services are converted into the standard file by setting different conversion rules, cross-service use cannot be achieved, and concurrent writing of big data is achieved by concurrently writing the HIVE original table data into the ES partition in batches, so that the data conversion efficiency is improved. And the data writing efficiency of writing the HIVE data into the ES is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and computer-readable storage medium. Background Art

[0002] In the Internet industry, data processing is a vital task because a large amount of information and data is generated every day. These data need to be effectively stored, managed, and processed in order to extract valuable information, drive business decisions, and optimize operations. Various data storage technologies are involved, each with its specific application scenarios and advantages. For example, HIVE, as a data warehouse tool based on Hadoop, is particularly suitable for processing and analyzing large-scale data. Redis, as an in-memory data structure storage system, has the advantages of high-speed read and write performance and caching capabilities for high-frequency data. Elasticsearch (ES) is a powerful distributed search engine. ES excels in real-time full-text search and analysis, and is particularly suitable for data storage and query in single business scenarios.

[0003] However, different data storage methods often serve different business needs, involving the issue of data exchange between different systems. For example, HIVE may have completed large-scale data cleaning and preprocessing and generated valuable results, but in order to achieve fast query and real-time application, this data may need to be synchronized to Redis for caching, or indexed to Elasticsearch (ES) to support efficient search.

[0004] The solution for writing HIVE data to Elasticsearch (ES) usually involves the ETL (extraction, transformation, loading) process of data. After setting up the index and mapping of ES, extract the required data from the Hive table through Hive SQL query, and then perform some preprocessing or transformation on the data according to the index structure of ES. Finally, use Hadoop's sqoop tool or data flow tools such as Logstash and Kafka to import data from Hive to ES. However, the way HIVE data is written to ES is usually very business-customized and cannot be used across businesses. If the amount of HIVE data is very large and needs to be imported in batches, etc., when problems occur when writing to ES, manual adjustments and retries are required, which affects the efficiency of writing HIVE data to ES.

[0005] Therefore, how to improve the efficiency of writing HIVE data into ES is a problem that needs to be solved urgently. Summary of the invention

[0006] The main purpose of this application is to provide a data processing method, device and computer-readable storage medium, aiming to solve the technical problem of how to improve the data writing efficiency of HIVE data into ES.

[0007] To achieve the above object, the present application provides a data processing method, which comprises the following steps:

[0008] Based on HIVE original table data and sftp files, determine standard files, wherein the standard files include XML files, DAT files and CTL files;

[0009] Polling the standard file to determine whether the CTL file is ready;

[0010] If the CTL file is ready, the DAT file is parsed based on the XML file to obtain a target parsed file;

[0011] The target parsed files are concurrently written in batches to the ES partitions.

[0012] Furthermore, if the CTL file is ready, the step of parsing the DAT file based on the XML file to obtain a target parsed file includes:

[0013] If the CTL file is ready, obtain the ES index configuration information corresponding to the standard file;

[0014] The DAT file is parsed based on the ES index configuration information and the XML file to obtain the target parsed file.

[0015] Furthermore, after the step of concurrently writing the target parsed files into the ES partition in batches, the data processing method further includes:

[0016] Obtain the writing progress corresponding to the target parsed file in real time, and update the writing progress to the MySQL table;

[0017] Obtaining the writing progress in the MySQL table, as well as the writing time and writing rate corresponding to the target parsed file in real time;

[0018] The write progress, write time, and write rate are displayed in the monitoring table.

[0019] Furthermore, after the step of displaying the writing progress, writing time and writing rate in the monitoring table, the data processing method further includes:

[0020] When abnormal information corresponding to the write operation is monitored, the abnormal information is displayed in the monitoring table;

[0021] If an interruption instruction corresponding to the abnormal information is received, the write operation is interrupted, and the target parsed file corresponding to the write operation is cleared in the ES partition;

[0022] When the target parsing file is cleaned up, the step of returning to the step of parsing the DAT file based on the configuration information and the XML file to obtain the target parsing file is performed.

[0023] Furthermore, after the step of concurrently writing the target parsed files into the ES partition in batches, the data processing method further includes:

[0024] When the writing of the target parsed file is completed, determine whether the amount of written data corresponding to the ES partition is consistent with the amount of data of the target parsed file, and perform sampling verification on the data written to the ES partition based on the sftp file;

[0025] If the amount of written data corresponding to the ES partition is inconsistent with the amount of data in the target parsed file, or the data sampling verification written to the ES partition fails, it is determined that the data verification has failed, and an alarm message is output;

[0026] If the amount of written data corresponding to the ES partition is consistent with the amount of data in the target parsed file, and the data sampling verification written to the ES partition passes, the index data version corresponding to the ES partition is recorded.

[0027] Furthermore, after the step of concurrently writing the target parsed files into the ES partition in batches, the data processing method further includes:

[0028] Poll the index data versions corresponding to the ES data in the ES partition to determine whether the index data versions corresponding to the ES data have been updated;

[0029] If the update of each index data version corresponding to the ES data is completed, the ES data corresponding to the ES partition is switched to the ES data corresponding to the updated index data version;

[0030] If the update of each index data version corresponding to the ES data is not completed, an alarm message will be output when the delay time is reached at the current moment.

[0031] Furthermore, the step of determining the standard file based on HIVE original table data and sftp file includes:

[0032] Obtain HIVE original table data and convert HIVE original table data into HIVE result table data;

[0033] The HIVE result table data is extracted into an sftp file to obtain the standard file, wherein the standard file includes an XML file, a DAT file, and a CTL file.

[0034] Furthermore, the step of extracting HIVE result table data into an sftp file to obtain the standard file includes:

[0035] Extract the HIVE result table data into an sftp file, and generate the XML file based on the field order and field type of the HIVE result table data;

[0036] The DAT file is generated based on HIVE result table data, and the CTL file is generated based on the number of records and encoding format of the DAT file.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a data processing device, which includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor, and the data processing program implements the steps of the aforementioned data processing method when executed by the processor.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a data processing program is stored, and when the data processing program is executed by a processor, the steps of the aforementioned data processing method are implemented.

[0039] The present application determines a standard file based on HIVE original table data and an sftp file, wherein the standard file includes an XML file, a DAT file, and a CTL file; then polls the standard file to determine whether the CTL file is ready; and then if the CTL file is ready, parses the DAT file based on the XML file to obtain a target parsed file; then concurrently batch writes the target parsed file to an ES partition, converts HIVE original table data into a standard file, parses the DAT file in the standard file to obtain the target parsed file, and concurrently batch writes the target parsed file to the ES partition, sets different conversion rules to convert HIVE original table data of different businesses into standard files, and achieves the inability to use across businesses, and concurrently writes large data into ES by concurrently batch writing to ES partitions, thereby improving the data writing efficiency of HIVE data into ES. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0043] Figure 2 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application.

[0044] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0047] The main solution of this application is: based on HIVE original table data and sftp files, determine the standard file, wherein the standard file includes XML file, DAT file and CTL file; poll the standard file to determine whether the CTL file is ready; if the CTL file is ready, parse the DAT file based on the XML file to obtain the target parsed file; and concurrently write the target parsed file into ES partitions in batches.

[0048] Currently, data processing is a vital task in the Internet industry, as a large amount of information and data is generated every day. These data need to be effectively stored, managed, and processed in order to extract valuable information, drive business decisions, and optimize operations. This involves a variety of different data storage technologies, each of which has its specific application scenarios and advantages. For example, HIVE, as a data warehouse tool based on Hadoop, is particularly suitable for processing and analyzing large-scale data. Redis, as an in-memory data structure storage system, has the advantages of high-speed read and write performance and caching capabilities for high-frequency data. Elasticsearch (ES) is a powerful distributed search engine. ES excels in real-time full-text search and analysis, and is particularly suitable for data storage and query in single business scenarios.

[0049] However, different data storage methods often serve different business needs, involving the issue of data exchange between different systems. For example, HIVE may have completed large-scale data cleaning and preprocessing and generated valuable results, but in order to achieve fast query and real-time application, this data may need to be synchronized to Redis for caching, or indexed to Elasticsearch (ES) to support efficient search.

[0050] The solution for writing HIVE data to Elasticsearch (ES) usually involves the ETL (extraction, transformation, loading) process of data. After setting up the index and mapping of ES, extract the required data from the Hive table through Hive SQL query, and then perform some preprocessing or conversion on the data according to the index structure of ES. Finally, use Hadoop's sqoop tool or data flow tools such as Logstash and Kafka to import data from Hive to ES. However, the way HIVE data is written to ES is usually very business-customized and cannot be used across businesses. If the amount of HIVE data is very large and needs to be imported in batches, etc., when problems occur in writing to ES, manual adjustments and retries are required, which affects the efficiency of writing HIVE data to ES. Therefore, how to improve the efficiency of writing HIVE data to ES is a problem that needs to be solved urgently.

[0051] This application converts HIVE original table data into standard files, parses the DAT file in the standard file to obtain the target parsed file, and concurrently writes the target parsed file into ES partitions in batches. By setting different conversion rules, HIVE original table data of different businesses are converted into standard files, which cannot be used across businesses. Concurrent writing of big data into ES partitions is achieved by concurrent batch writing, which improves the data writing efficiency of HIVE data into ES.

[0052] It should be noted that the execution subject of this embodiment may be a data processing device, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a data processing device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the data processing device as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, this application proposes a data processing method of the first embodiment, please refer to Figure 1 , the data processing method includes steps S101 to S104:

[0054] Step S101, determining a standard file based on HIVE original table data and an sftp file, wherein the standard file includes an XML file, a DAT file, and a CTL file;

[0055] In this embodiment, for HIVE original table data, conversion rules for converting HIVE original table data into standard files are pre-configured. For example, a field type mapping rule is configured to correspond the fields in the standard file to the field types of the Elasticsearch (ES) index, including the setting of the ES index ID, the parsing of the ES nested type and the array type field, to ensure that the data can be correctly stored and retrieved according to the preset rules when imported into ES.

[0056] In this embodiment, after obtaining HIVE original table data, a standard file is determined based on HIVE original table data and sftp file, that is, HIVE original table data is extracted into a preset configured sftp file and saved as a standard file. Further, in a feasible implementation manner, step S101 may include steps S1011 to S1012:

[0057] Step S1011, obtaining HIVE original table data, and converting the HIVE original table data into HIVE result table data;

[0058] Step S1012, extracting HIVE result table data into an sftp file to obtain the standard file, wherein the standard file includes an XML file, a DAT file, and a CTL file.

[0059] In this embodiment, after the HIVE original table data is acquired, the HIVE original table data is converted into HIVE result table data. Specifically, the HIVE original table data is converted into HIVE result table data according to relevant technologies.

[0060] After obtaining the HIVE result table data, extract the HIVE result table data into an sftp file to obtain the standard file, that is, extract the HIVE result table data into a preset configured sftp file and save it as a standard file. Further, in a feasible implementation manner, step S1012 may include steps a1 to a2:

[0061] Step a1, extracting HIVE result table data into an sftp file, and generating the XML file based on the field order and field type of the HIVE result table data;

[0062] Step a2, generating the DAT file based on HIVE result table data, and generating the CTL file based on the number of records and encoding format of the DAT file.

[0063] In this embodiment, after the HIVE result table data is obtained, the HIVE result table data is extracted into an sftp file, and the XML file is generated based on the field order and field type of the HIVE result table data, that is, the field order and field type of the HIVE result table data are recorded through the XML file.

[0064] Next, a DAT file is generated based on the HIVE result table data, that is, the data of the preset rules in the HIVE result table data is stored in the DAT file by row. After obtaining the DAT file, a CTL file is generated based on the number of records and encoding format of the DAT file, that is, the number of records in the DAT file, encoding format and other information are saved through the CTL file, thereby realizing the conversion of the HIVE original table data into a standard file in a standard file format.

[0065] Step S102, polling the standard file to determine whether the CTL file is ready;

[0066] In this embodiment, after obtaining the standard file corresponding to the HIVE original table data, the standard file is polled to determine whether the CTL file is ready. Specifically, a scheduled task can be set to poll the standard file through the scheduled task to determine whether the CTL file is ready.

[0067] Step S103, if the CTL file is ready, the DAT file is parsed based on the XML file to obtain a target parsed file;

[0068] In this embodiment, if the CTL file is ready, the DAT file is parsed based on the XML file to obtain the target parsed file; that is, the DAT file is parsed to obtain the target parsed file through the information such as the number of DAT file records and the encoding format saved in the CTL file. Further, in a feasible implementation manner, step S103 may include steps S1031 to S1032:

[0069] Step S1031: if the CTL file is ready, obtain the ES index configuration information corresponding to the standard file;

[0070] Step S1032: Parse the DAT file based on the ES index configuration information and the XML file to obtain the target parsed file.

[0071] In this embodiment, if the CTL file is ready, the ES index configuration information corresponding to the standard file is obtained, wherein the ES index configuration information includes field mapping relationship, write ES time, write ES concurrency, monitoring rules, alarm contact, write speed and other information.

[0072] After obtaining the ES index configuration information, the DAT file is parsed based on the ES index configuration information and the XML file to obtain the target parsed file. That is, according to the field mapping relationship in the ES index configuration information, the number of DAT file records, encoding format and other information saved in the CTL file, the DAT file is parsed to obtain the target parsed file to improve the accuracy of the target parsed file.

[0073] Step S104, writing the target parsed files into ES partitions concurrently in batches.

[0074] In this embodiment, after the target parsed files are obtained, the target parsed files are concurrently written in batches to the ES partitions to write HIVE data into the ES.

[0075] In a feasible implementation manner, after step S104, the data processing method may further include steps b1 to b3:

[0076] Step b1, when the writing of the target parsed file is completed, determining whether the amount of written data corresponding to the ES partition is consistent with the amount of data of the target parsed file, and performing sampling verification on the data written to the ES partition based on the sftp file;

[0077] Step b2: if the amount of written data corresponding to the ES partition is inconsistent with the amount of data in the target parsed file, or the data sampling verification written to the ES partition fails, it is determined that the data verification fails, and an alarm message is output;

[0078] Step b3: If the amount of written data corresponding to the ES partition is consistent with the amount of data in the target parsed file, and the data sampling verification written to the ES partition passes, then the index data version corresponding to the ES partition is recorded.

[0079] In this embodiment, when the writing of the target parsed file is completed, it is determined whether the amount of written data corresponding to the ES partition is consistent with the amount of data of the target parsed file. Specifically, the amount of data of the target parsed file and the amount of written data of the data written in the ES partition can be obtained, and the amount of data of the target parsed file is compared with the amount of written data to determine whether the amount of written data is consistent with the amount of data of the target parsed file. At the same time, the data written to the ES partition is sampled and verified based on the sftp file. Specifically, the data written to the sftp file and the ES partition are sampled and compared respectively to determine whether the data written to the sftp file and the ES partition are consistent, that is, the consistency of the data written to the sftp file and the ES partition is verified by sampling.

[0080] If the amount of written data corresponding to the ES partition is consistent with the amount of data in the target parsed file, and the data sampling verification written to the ES partition fails, the index data version corresponding to the ES partition is recorded.

[0081] If the amount of written data corresponding to the ES partition is inconsistent with the data amount of the target parsed file, or the data sampling verification written to the ES partition passes, the data verification is determined to have failed, and an alarm message is output, for example, the corresponding alarm message is output to the alarm contact person to ensure the accuracy of the data written in the ES partition.

[0082] In a feasible implementation manner, after step S104, the data processing method may further include steps c1 to c3:

[0083] Step c1, polling the index data versions corresponding to the ES data in the ES partition to determine whether each index data version corresponding to the ES data has been updated;

[0084] Step c2: if the update of each index data version corresponding to the ES data is completed, the ES data corresponding to the ES partition is switched to the ES data corresponding to the updated index data version;

[0085] Step c3: If the update of each index data version corresponding to the ES data is not completed, an alarm message is output when the delay time is reached at the current moment.

[0086] In this embodiment, when the writing of the target parsed file is completed, the index data version corresponding to the ES data that has been written in the ES partition is polled to determine whether the index data version corresponding to the ES data that has been written has been updated, that is, whether the index data versions corresponding to all ES data in the ES partition have been updated, so as to determine whether the index data versions of all ES data in the ES partition are consistent, so as to ensure the version time consistency of the related business data.

[0087] If the update of each index data version corresponding to the ES data is completed, the ES data corresponding to the ES partition will be switched to the ES data corresponding to the updated index data version; that is, the new version of ES data will be uniformly switched to ensure the consistency of business data version.

[0088] If the update of each index data version corresponding to the ES data is not completed, determine whether it exceeds the set delay time. If it exceeds the set delay time, for example, when the delay time is reached at the current moment, output an alarm message, for example, output the corresponding alarm message to the alarm contact.

[0089] It should be noted that you can also view the ES index version of the written data through index management, and manually adjust the index version switch to ensure the consistency of business data version.

[0090] The data processing method proposed in this embodiment determines a standard file based on HIVE original table data and an sftp file, wherein the standard file includes an XML file, a DAT file, and a CTL file; then polls the standard file to determine whether the CTL file is ready; and then if the CTL file is ready, parses the DAT file based on the XML file to obtain a target parsed file; then concurrently batch writes the target parsed file to the ES partition, converts HIVE original table data into a standard file, parses the DAT file in the standard file to obtain the target parsed file, and concurrently batch writes the target parsed file to the ES partition, sets different conversion rules to convert HIVE original table data of different businesses into standard files, and implements the inability to use across businesses, and implements concurrent writing of large data by concurrent batch writing to the ES partition, thereby improving the data writing efficiency of HIVE data to the ES.

[0091] In a feasible implementation manner, after step S104, the data processing method may further include steps S201 to S202:

[0092] Step S201, obtaining the writing progress corresponding to the target parsed file in real time, and updating the writing progress to the MySQL table;

[0093] Step S202, obtaining the writing progress in the MySQL table, and the writing time and writing rate corresponding to the target parsed file in real time;

[0094] Step S203, displaying the writing progress, writing time and writing rate in the monitoring table.

[0095] In this embodiment, in the process of concurrently writing the target parsed files into the ES partition in batches, the writing progress corresponding to the target parsed files is obtained in real time, and the writing progress is updated to the MySQL table to facilitate the monitoring of the writing progress of the ES partition writing data.

[0096] Next, in the process of concurrently writing the target parsed files in batches to the ES partition, the writing progress in the MySQL table, as well as the writing time and writing rate corresponding to the target parsed files can be obtained in real time. For example, the writing time can be the time difference between the current moment and the start moment of writing the target parsed files concurrently in batches to the ES partition. The writing rate can be determined by the amount of written data in the ES partition and the time difference.

[0097] After obtaining the write progress, write time, and write rate, they are displayed in the monitoring table so that users can detect write anomalies in a timely manner based on the displayed content, further improving the data writing efficiency of HIVE data into ES.

[0098] Furthermore, in a feasible implementation manner, after step S203, the data processing method may further include steps S204 to S206:

[0099] Step S204, when abnormal information corresponding to the write operation is monitored, the abnormal information is displayed in the monitoring table;

[0100] Step S205: if an interruption instruction corresponding to the abnormal information is received, the write operation is interrupted, and the target parsed file corresponding to the write operation is cleared in the ES partition;

[0101] Step S206, when the target parsing file is cleaned up, return to the step of parsing the DAT file based on the configuration information and the XML file to obtain the target parsing file.

[0102] In this embodiment, during the writing process of the target parsed file, if there is a parsing error or a writing error, it is determined that the abnormal information corresponding to the write operation is monitored. At this time, the abnormal information is displayed in the monitoring table for the user to view. When the user views the abnormal information, he can choose to interrupt the current write operation and try again.

[0103] If an interrupt instruction corresponding to the exception information is received, the write operation is interrupted, that is, the operation of concurrently writing the target parsed files to the ES partition in batches is interrupted, and the target parsed files corresponding to the write operation are cleaned up in the ES partition; and when the target parsed files are cleaned up, return to execute the step of parsing the DAT file based on the configuration information and the XML file to obtain the target parsed file, so as to achieve retry when writing fails, thereby further improving the data writing efficiency of HIVE data writing to ES.

[0104] The data processing method proposed in this embodiment obtains the writing progress corresponding to the target parsed file in real time and updates the writing progress to the MySQL table; then obtains the writing progress in the MySQL table, as well as the writing time and writing rate corresponding to the target parsed file in real time; displays the writing progress, writing time and writing rate in the monitoring table to facilitate timely detection of writing anomalies, thereby further improving the data writing efficiency of HIVE data writing to ES.

[0105] The data processing device provided in the embodiment of the present application adopts the data processing method in the above embodiment, which can solve the technical problem of how to improve the data writing efficiency of HIVE data into ES. Compared with the prior art, the beneficial effects of the data processing device provided in the embodiment of the present application are the same as the beneficial effects of the data processing method provided in the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0106] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0107] Reference below Figure 2 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 2 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0108] like Figure 2As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the data processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0109] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0110] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of how to improve the data writing efficiency of HIVE data into ES. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0111] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0112] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0113] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0114] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0115] The computer-readable storage medium may be included in the data processing device; or may exist independently without being installed in the data processing device.

[0116] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data processing device, the data processing device: determines a standard file based on HIVE original table data and an sftp file, wherein the standard file includes an XML file, a DAT file, and a CTL file; polls the standard file to determine whether the CTL file is ready; if the CTL file is ready, parses the DAT file based on the XML file to obtain a target parsed file; and concurrently writes the target parsed file to an ES partition in batches.

[0117] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0120] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of how to improve the data writing efficiency of writing HIVE data to ES. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0121] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0122] The computer program product provided in this application can solve the technical problem of how to improve the efficiency of writing HIVE data into ES. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the data processing method provided in the above embodiment, which will not be repeated here.

[0123] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises the following steps: Based on HIVE original table data and sftp files, determine standard files, wherein the standard files include XML files, DAT files and CTL files; Polling the standard file to determine whether the CTL file is ready; If the CTL file is ready, the DAT file is parsed based on the XML file to obtain a target parsed file; The target parsed files are concurrently written in batches to the ES partitions.

2. The data processing method according to claim 1, characterized in that: If the CTL file is ready, the step of parsing the DAT file based on the XML file to obtain a target parsed file includes: If the CTL file is ready, obtain the ES index configuration information corresponding to the standard file; The DAT file is parsed based on the ES index configuration information and the XML file to obtain the target parsed file.

3. The data processing method according to claim 1, characterized in that: After the step of concurrently writing the target parsed files into the ES partition in batches, the data processing method further includes: Obtain the writing progress corresponding to the target parsed file in real time, and update the writing progress to the MySQL table; Obtaining the writing progress in the MySQL table, as well as the writing time and writing rate corresponding to the target parsed file in real time; The write progress, write time, and write rate are displayed in the monitoring table.

4. The data processing method according to claim 3, characterized in that: After the step of displaying the writing progress, writing time and writing rate in the monitoring table, the data processing method further includes: When abnormal information corresponding to the write operation is monitored, the abnormal information is displayed in the monitoring table; If an interruption instruction corresponding to the abnormal information is received, the write operation is interrupted, and the target parsed file corresponding to the write operation is cleared in the ES partition; When the target parsing file is cleaned up, the step of returning to the step of parsing the DAT file based on the configuration information and the XML file to obtain the target parsing file is performed.

5. The data processing method according to claim 1, characterized in that: After the step of concurrently writing the target parsed files into the ES partition in batches, the data processing method further includes: When the writing of the target parsed file is completed, determine whether the amount of written data corresponding to the ES partition is consistent with the amount of data of the target parsed file, and perform sampling verification on the data written to the ES partition based on the sftp file; If the amount of written data corresponding to the ES partition is inconsistent with the amount of data in the target parsed file, or the data sampling verification written to the ES partition fails, it is determined that the data verification has failed, and an alarm message is output; If the amount of written data corresponding to the ES partition is consistent with the amount of data in the target parsed file, and the data sampling verification written to the ES partition passes, the index data version corresponding to the ES partition is recorded.

6. The data processing method according to claim 1, characterized in that: After the step of concurrently writing the target parsed files into the ES partition in batches, the data processing method further includes: Poll the index data versions corresponding to the ES data in the ES partition to determine whether the index data versions corresponding to the ES data have been updated; If the update of each index data version corresponding to the ES data is completed, the ES data corresponding to the ES partition is switched to the ES data corresponding to the updated index data version; If the update of each index data version corresponding to the ES data is not completed, an alarm message will be output when the delay time is reached at the current moment.

7. The data processing method according to any one of claims 1 to 6, characterized in that: The step of determining the standard file based on HIVE original table data and sftp file includes: Obtain HIVE original table data and convert HIVE original table data into HIVE result table data; The HIVE result table data is extracted into an sftp file to obtain the standard file, wherein the standard file includes an XML file, a DAT file, and a CTL file.

8. The data processing method according to claim 7, characterized in that: The step of extracting HIVE result table data into an sftp file to obtain the standard file comprises: Extract the HIVE result table data into an sftp file, and generate the XML file based on the field order and field type of the HIVE result table data; The DAT file is generated based on HIVE result table data, and the CTL file is generated based on the number of records and encoding format of the DAT file.

9. A data processing device, characterized in that: The data processing device comprises: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program implements the steps of the data processing method according to any one of claims 1 to 8 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data processing program, and when the data processing program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

Images