Data file processing method, system and equipment, medium and program product

Abstract

The invention provides a data file processing method, system and device, a medium and a program product, and the processing method comprises the steps: monitoring whether a data file exists in a data directory or not in real time, and if yes, decompressing the data file; and uploading the decompressed data file to a distributed file system in real time. According to the method, whether the data file exists in the data directory or not is monitored in real time, and if yes, the data file is decompressed; and the decompressed data file is uploaded to the distributed file system in real time, so that the use efficiency of the server is improved, and the resource consumption of the server is reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data file processing, in particular to a data file processing method, system, device, medium and program product. BACKGROUND

[0002] At present, in the case of batch mode timing uploading data files to big data HDFS (distributed file system), when batch data files are uploaded to big data, the data files or compressed package files are first pushed to the specified directory, and the timing calling program uploads the files to big data HDFS.

[0003] When the existing program uploads multiple data files to the big data environment at the same time, it will cause too much consumption of server resources, and in extreme cases, uploading files will cause interruption of other application programs and other problems; and when there is no data package to upload, part of the server resources is in an idle state and cannot be fully utilized. SUMMARY

[0004] The technical problem to be solved by the present disclosure is to overcome the defects of large consumption of server resources and low use efficiency in the prior art by adopting the method of timing uploading data files to big data HDFS, and to provide a data file processing method, system, device, medium and program product.

[0005] The present disclosure solves the above technical problems by the following technical solutions:

[0006] The first aspect of the present disclosure provides a data file processing method, the processing method comprising:

[0007] Real-time monitoring whether there is a data file in the data directory, if yes, decompressing the data file;

[0008] Real-time uploading the decompressed data file to a distributed file system.

[0009] Preferably, the processing method further comprises:

[0010] Collecting data files from a source database;

[0011] Transferring the data files to a data directory.

[0012] Preferably, the processing method further comprises:

[0013] Obtaining log information of uploading the decompressed data file to the distributed file system;

[0014] Storing the log information into a database.

[0015] Preferably, the processing method further comprises:

[0016] packaging the application program into a program package;

[0017] deploying the program package into an application server.

[0018] Preferably, the processing method further comprises:

[0019] If there is no data file in the data directory, the step of monitoring whether there is a data file in the data directory in real time is continuously executed.

[0020] Preferably, the source database comprises at least one of a distributed database and a relational database.

[0021] The second aspect of the present disclosure provides a processing system of a data file, the processing system comprising:

[0022] a monitoring module for monitoring whether there is a data file in a data directory in real time, and calling a decompression module if there is a data file;

[0023] the decompression module is configured to decompress the data file;

[0024] an uploading module for uploading the decompressed data file to a distributed file system in real time.

[0025] Preferably, the processing system further comprises:

[0026] a collection module for collecting a data file from a source database;

[0027] a transmission module for transmitting the data file to a data directory.

[0028] Preferably, the processing system further comprises:

[0029] an acquisition module for acquiring log information of uploading the decompressed data file to the distributed file system;

[0030] a storage module for storing the log information in a database.

[0031] Preferably, the processing system further comprises:

[0032] a packaging module for packaging an application program into a program package;

[0033] a deployment module for deploying the program package into an application server.

[0034] Preferably, the processing system further comprises:

[0035] a monitoring module for, if there is no data file in the data directory, continuously executing the step of monitoring whether there is a data file in the data directory in real time.

[0036] Preferably, the source database comprises at least one of a distributed database and a relational database.

[0037] The third aspect of the present disclosure provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and configured to run on the processor, wherein the processor executes the computer program to implement the data file processing method of the first aspect.

[0038] The fourth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the data file processing method of the first aspect.

[0039] The fifth aspect of the present disclosure provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the data file processing method of the first aspect.

[0040] On the basis of common knowledge in the art, the above-mentioned preferred conditions can be combined in any manner to obtain preferred examples of the present disclosure.

[0041] The positive progress effect of the present disclosure is that:

[0042] The present disclosure improves the use efficiency of the server and reduces the consumption of resources of the server by monitoring whether there is a data file in the data directory in real time, and decompressing the data file if there is one, and uploading the decompressed data file to the distributed file system in real time. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The flowchart of the data file processing method provided for the first embodiment of the present disclosure is shown in FIG. 1.

[0044] Figure 2 The module schematic diagram of the data file processing system provided for the second embodiment of the present disclosure is shown in FIG. 2.

[0045] Figure 3 The structure schematic diagram of the electronic device for implementing the data file processing method of the third embodiment of the present disclosure is shown in FIG. 3. DETAILED DESCRIPTION

[0046] The present disclosure will be further described by way of examples below, but the present disclosure is not limited in the scope of the examples.

[0047] The prefix words such as "first", "second" in the embodiments of the present disclosure are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of ordinal words and other prefix words in the embodiments of the present disclosure to distinguish the described objects does not constitute a limitation on the described objects, and the description of the described objects is referred to the context in the claims or embodiments

[0048] The description, and should not constitute redundant limitations because of the use of such prefix words. In addition, in the description of the present embodiment, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0049] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present disclosure comply with relevant laws and regulations and do not violate public order and good customs.

[0050] Embodiment 1

[0051] Figure 1 A flowchart of a data file processing method provided in Embodiment 1 of the present disclosure is shown in Figure 1 The processing method comprises the following steps:

[0052] Step 101, real-time monitoring whether there is a data file in the data directory, if yes, executing step 102;

[0053] In the present embodiment, the program for real-time monitoring the data directory is written by Java language, specifically, the application program (i.e. Java application program) written by Java language is used to real-time monitor the data directory.

[0054] Step 102, decompressing the data file;

[0055] Step 103, real-time uploading the decompressed data file to the distributed file system.

[0056] In the present embodiment, the application program written by Java language decompresses the compressed package data file pushed by the ETL (collection tool) tool and uploads it to the big data HDFS, and writes the generated log data into the database. Specifically, the application program written by Java language uploads the data file to the HDFS through the API of Hadoop, and further writes the decompressed data file into the hive (Hadoop-based data warehouse tool) or hudi (Hadoop-based data lake management framework) of HDFS, so as to facilitate subsequent analysis of the written data by using SQL through the hive or hudi.

[0057] It should be noted that the Java application program decompresses all the compressed package data files under the permission of the thread pool.

[0058] In addition, the Java application has the permission to access the data directory where the data file is stored by pushing the data file to the data directory under real-time monitoring through the ETL tool.

[0059] In an optional embodiment, the processing method further comprises:

[0060] Step 201, collecting a data file from a source database;

[0061] In an optional embodiment, the source database comprises at least one of a distributed database and a relational database.

[0062] The relational database comprises Oracle (a relational database management system), Mysql (a relational database management system), Sqlserver (a relational database management system), Dameng database, etc.

[0063] Step 202, transmitting the data file to a data directory.

[0064] In this embodiment, the ETL tool kettle is used to collect the data file from the source database (for example, an Oracle database or other databases) to the specified data directory, so as to ensure that the data file is not lost.

[0065] In an optional embodiment, the processing method further comprises:

[0066] Step 301, obtaining log information about uploading the decompressed data file to the distributed file system;

[0067] Step 302, storing the log information in a database.

[0068] In this embodiment, for example, a log table is established in the Mysql database as the source database, which is used to store the log information about uploading the decompressed data file to the distributed file system by the Java application, so as to provide a reference basis and a state for subsequent calculation, and also facilitate the later use, maintenance and viewing.

[0069] It should be noted that in the database, the log table is established according to the business logic, and the data file received by the Java application and the log information about uploading the data file are written into the database.

[0070] This embodiment realizes real-time monitoring of the data directory by the Java application, real-time uploading of the data file to the HDFS, and effective use of the application server resources, thereby improving the use efficiency of the application server, reducing the sharp consumption of the resources of the application server in a short time, and writing the log information about the uploaded data file into the database.

[0071] In an optional embodiment, the processing method further comprises:

[0072] Step 401, packaging the written application program into a program package;

[0073] Step 402, deploying the program package into an application server.

[0074] In this embodiment, the written Java application program code is packaged by an ETL tool, the program package is deployed onto an application server and started to monitor a specified data directory.

[0075] It should be noted that the Java application program is continuously running on the application server.

[0076] In an optional embodiment, the processing method further comprises:

[0077] If there is no data file in the real-time monitored data directory, step 101 is continuously executed.

[0078] In the specific implementation process, for example, taking a Mysql database as a source database, data files are collected from the source database (for example, an Oracle database or other databases) into a specified data directory by an ETL tool kettle; taking the Mysql database as a source database, a log table is established in the database to store log information of the Java language written application program uploading the decompressed data file into the distributed file system, which provides a reference basis and state for subsequent calculation; a big data environment is installed and tested to enable the big data to normally store data files (specifically, corresponding HDFS, yarn and other middleware are installed in the big data environment); the written Java application program code is packaged, the program package is deployed onto an application server and started to monitor the specified data directory; after the Java application program is started, the data files pushed by the ETL tool into the specified data directory are monitored in real time, the data files are decompressed and uploaded to the big data environment.

[0079] It should be noted that the corresponding middleware of the big data is installed to enable the big data environment to normally run and store HDFS data.

[0080] In this embodiment, whether there is a data file in the data directory is monitored in real time, if there is, the data file is decompressed; the decompressed data file is uploaded to the distributed file system in real time, which improves the use efficiency of the server and reduces the consumption of resources of the server.

[0081] Embodiment 2

[0082] Corresponding to the foregoing data file processing method embodiment, the disclosure also provides an embodiment of a data file processing system.

[0083] Figure 2 A module schematic diagram of a data file processing system provided for Embodiment 2 of the present disclosure is shown in Figure 2 The processing system comprises a monitoring module 21, a decompression module 22, and an uploading module 23.

[0084] The monitoring module 21 is configured to monitor whether there is a data file in the data directory in real time, and if so, call the decompression module 22.

[0085] In this embodiment, the program for monitoring the data directory in real time is written in Java language. Specifically, an application program (i.e., Java application program) written in Java language monitors the data directory in real time.

[0086] The decompression module 22 is configured to decompress the data file.

[0087] The uploading module 23 is configured to upload the decompressed data file to the distributed file system.

[0088] In this embodiment, the application program written in Java language decompresses the compressed package data file pushed by the ETL tool and uploads it to the big data HDFS, and writes the generated log data into the database. Specifically, the application program written in Java language uploads the data file to the HDFS through the API of Hadoop, and further writes the decompressed data file into the hive or hudi of HDFS, so as to facilitate subsequent analysis of the written data by using SQL through the hive or hudi.

[0089] It should be noted that the Java application program decompresses all the compressed package data files under the permission of the thread pool.

[0090] In addition, the data file is pushed to the data directory monitored in real time by the ETL tool, and the Java application program has the permission to access the data directory where the data file is stored.

[0091] In an optional embodiment, as shown in Figure 2 The processing system further comprises a collection module 24 and a transmission module 25.

[0092] The collection module 24 is configured to collect data files from a source database.

[0093] In an optional embodiment, the source database comprises at least one of a distributed database and a relational database.

[0094] The relational database includes Oracle (a relational database management system), Mysql (a relational database management system), Sqlserver (a relational database management system), Dameng database, and the like.

[0095] The transmission module 25 is configured to transmit the data file into the data directory.

[0096] In the embodiment, the ETL tool kettle is used to collect the data file from the source database (for example, an Oracle database or other database) into the designated data directory, so as to ensure that the data file is not lost.

[0097] In an optional embodiment, as shown in Figure 2 The processing system further includes an acquisition module 26 and a storage module 27.

[0098] The acquisition module 26 is configured to acquire log information about uploading the decompressed data file into the distributed file system.

[0099] The storage module 27 is configured to store the log information into the database.

[0100] In the embodiment, for example, the Mysql database is taken as the source database, and a log table is established in the database, which is used to store the log information about uploading the decompressed data file into the distributed file system by the application program written in Java, so as to provide a reference basis and a state for subsequent calculation, and facilitate the later use, maintenance and the like.

[0101] It should be noted that in the database, the log table is established according to the business logic, and the data file received by the Java application program and the log information about uploading the data file are written into the database.

[0102] The embodiment realizes real-time monitoring of the data directory by the Java application program, real-time uploading of the data file into the HDFS, and effective use of the application server resources, so as to improve the use efficiency of the application server, reduce the sharp consumption of the resources of the application server in a short time, and write the log information about the uploaded data file into the database.

[0103] In an optional embodiment, as shown in Figure 2 The processing system further includes a packaging module 28 and a deployment module 29.

[0104] The packaging module 28 is configured to package the written application program into a program package.

[0105] The deployment module 29 is configured to deploy the program package into the application server.

[0106] In this embodiment, the written Java application code is packaged by an ETL tool, the program package is deployed to an application server and started to monitor the specified data directory.

[0107] It should be noted that the Java application is continuously running on the application server.

[0108] In an optional embodiment, the monitoring module 21 is configured to, if no data file is found in the real-time monitoring data directory, continue to perform the step of monitoring whether there is a data file in the real-time monitoring data directory.

[0109] In the specific implementation process, for example, taking a Mysql database as a source database, data files are collected from the source database (for example, an Oracle database or other databases) to a specified data directory by an ETL tool kettle; taking a Mysql database as a source database, a log table is established in the database to store log information of the Java application uploading the decompressed data file to the distributed file system, which provides a reference basis and state for subsequent calculation; a big data environment is installed and tested to enable the big data to normally store data files (specifically, corresponding HDFS, yarn and other middleware are installed in the big data environment); the written Java application code is packaged, the program package is deployed to an application server and started to monitor the specified data directory; after the Java application is started, the data files pushed to the specified data directory by the ETL tool are monitored in real time, the data files are decompressed and uploaded to the big data environment.

[0110] It should be noted that the corresponding middleware of the big data is installed to enable the big data environment to normally run and store HDFS data.

[0111] The embodiment realizes real-time monitoring of whether there is a data file in the data directory, and if there is, the data file is decompressed; the decompressed data file is uploaded to the distributed file system in real time, which improves the use efficiency of the server and reduces the consumption of resources of the server.

[0112] For the system embodiment, since it basically corresponds to the method embodiment, the related parts are described in the part of the method embodiment. The system embodiment described above is only illustrative, and the units described as separate components can or can not be physically separated, and the components of the unit can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present disclosure.

[0113] Embodiment 3

[0114] Figure 3 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the data file processing method described in any of the above embodiments. Figure 3 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0115] like Figure 3 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0116] Bus 93 includes a data bus, an address bus, and a control bus.

[0117] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0118] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0119] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the data file processing method provided in any of the above embodiments.

[0120] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 96. Figure 3 As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0121] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into units / modules embodied by multiple units / modules.

[0122] Embodiment 4

[0123] The embodiment 4 of the present disclosure further provides a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the data file processing method provided by any of the above embodiments.

[0124] More specifically, the readable storage medium can include, but is not limited to, a portable disc, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0125] Embodiment 5

[0126] The embodiment 5 of the present disclosure further provides a computer program product, which includes a computer program. The computer program is executed by a processor to implement the data file processing method according to any of the above embodiments.

[0127] The program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0128] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an illustration, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, and such changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A method for processing data files, characterized in that, The processing method includes: The system monitors the data directory in real time to check for data files. If a data file is found, it is decompressed. The decompressed data files are uploaded to the distributed file system in real time.

2. The data file processing method as described in claim 1, characterized in that, The processing method further includes: Collect data files from the source database; Transfer the data file to the data directory.

3. The data file processing method as described in claim 1, characterized in that, The processing method further includes: Retrieve log information about uploading the decompressed data file to the distributed file system; The log information is stored in the database.

4. The data file processing method as described in claim 1, characterized in that, The processing method further includes: Package the written application into a package; Deploy the package to the application server.

5. The data file processing method as described in claim 1, characterized in that, The processing method further includes: If there are no data files in the real-time monitoring data directory, continue with the steps to check if there are any data files in the real-time monitoring data directory.

6. The data file processing method as described in claim 2, characterized in that, The source database includes at least one of a distributed database and a relational database.

7. A data file processing system, characterized in that, The processing system includes: The monitoring module is used to monitor in real time whether there are data files in the data directory. If so, it calls the decompression module. The decompression module is used to decompress the data file; The upload module is used to upload the decompressed data files to the distributed file system in real time.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the data file processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data file processing method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data file processing method as described in any one of claims 1 to 6.

Images