File processing method, device, storage medium and computer equipment

By obtaining and parsing the image files generated by the data warehouse tool nodes and combining them with directory information, the problem of waste of computing resources caused by small files during the data warehouse tool operation is solved, and automated file identification and processing is realized, which improves processing efficiency and System stability.

CN112231292BActive Publication Date: 2024-11-19HANGZHOU DT DREAM TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010930089.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-02-15
Publication Date
2024-11-19
Estimated Expiration
2039-02-15

AI Technical Summary

Technical Problem

Data warehouse tools generate a large number of small files during operation, resulting in wasted computing resources, and it is difficult for existing technologies to effectively process these files.

Method used

By obtaining the image files generated by the data warehouse tool node, combining the directory information to parse the original file information, and combining them according to preset rules to achieve automated identification and timely processing.

Benefits of technology

It effectively reduces the waste of computing resources, improves file processing efficiency, simplifies the process of obtaining original file information, and avoids the impact on the stability of the production environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112231292B_ABST
    Figure CN112231292B_ABST
Patent Text Reader

Abstract

The present invention proposes a file processing method, device, storage medium and computer equipment, wherein the file is a file in a data warehouse tool, the data warehouse tool includes a target type of node, and the method includes obtaining a mirror file generated by the target type of node; combining the directory information of the data warehouse tool, parsing the mirror file to obtain information of the original file to which the mirror file belongs; and merging the original file according to the information of the original file and combining preset rules. The present invention can realize automatic identification of files generated when the data warehouse tool is running, and merge the files in a timely manner.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a file processing method, device, storage medium and computer equipment. Background Art

[0002] Data warehouse tools can map structured data files into a database table and provide simple SQL (Structured Query Language) functions, which can convert SQL statements into distributed computing tasks for execution.

[0003] Data warehouse tools generally run on the Hadoop distributed file system, and a large number of small files are generated during the operation. Small files may be generated when the data source is imported into the data warehouse tool, or when the data table of the data warehouse tool is read for offline calculation. Usually, a single file needs to occupy a calculation process or thread when calculating. A large number of small files consumes more computing resources. Therefore, it is necessary to process the files generated during the operation of the data warehouse tool. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the purpose of the present invention is to provide a file processing method, device, storage medium and computer equipment, which can automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.

[0006] To achieve the above-mentioned purpose, a file processing method is proposed in an embodiment of the first aspect of the present invention, wherein the file is a file in a data warehouse tool, and the data warehouse tool includes a node of a target type, including: obtaining a mirror file generated by the node of the target type; parsing the mirror file in combination with the directory information of the data warehouse tool to obtain information of the original file to which the mirror file belongs; and merging the original file according to the information of the original file in combination with preset rules.

[0007] The file processing method proposed in the embodiment of the first aspect of the present invention obtains the image file generated by the node of the target type; parses the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merges the original files according to the information of the original files in combination with preset rules, thereby being able to automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.

[0008] To achieve the above-mentioned purpose, a file processing device is proposed in an embodiment of the second aspect of the present invention, wherein the file is a file in a data warehouse tool, and the data warehouse tool includes a node of a target type, including: an acquisition module, used to acquire the image file generated by the node of the target type; a parsing module, used to parse the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; and a merging processing module, used to merge the original file according to the information of the original file in combination with preset rules.

[0009] The file processing device proposed in the embodiment of the second aspect of the present invention obtains the image file generated by the node of the target type; combines the directory information of the data warehouse tool to parse the image file to obtain the information of the original file to which the image file belongs; merges the original file according to the information of the original file and combines it with preset rules, so as to automatically identify the files generated when the data warehouse tool is running, and merge the files in time.

[0010] To achieve the above-mentioned purpose, the non-temporary computer-readable storage medium proposed in the third aspect embodiment of the present invention, when the instructions in the storage medium are executed by the processor of the mobile terminal, enables the mobile terminal to execute a file processing method, the method comprising: the file processing method proposed in the first aspect embodiment of the present invention.

[0011] The non-temporary computer-readable storage medium proposed in the embodiment of the third aspect of the present invention obtains the image file generated by the node of the target type; parses the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merges the original files according to the information of the original file in combination with preset rules, thereby automatically identifying the files generated when the data warehouse tool is running, and merging the files in a timely manner.

[0012] To achieve the above-mentioned purpose, a computer program product proposed in an embodiment of the fourth aspect of the present invention, when the instructions in the computer program product are executed by a processor, executes a file processing method, the file is a file in a data warehouse tool, the data warehouse tool includes a node of a target type, and the method includes: obtaining a mirror file generated by the node of the target type; in combination with the directory information of the data warehouse tool, parsing the mirror file to obtain information of the original file to which the mirror file belongs; and merging the original files according to the information of the original files and in combination with preset rules.

[0013] The computer program product proposed in the fourth aspect of the embodiment of the present invention obtains the image file generated by the node of the target type; parses the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merges the original files according to the information of the original files in combination with preset rules, thereby being able to automatically identify the files generated when the data warehouse tool is running and merge the files in a timely manner.

[0014] The fifth aspect of the present invention also proposes a computer device, which includes a shell, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the shell, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the computer device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute: obtaining the image file generated by the node of the target type; parsing the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merging the original files according to the information of the original files in combination with preset rules.

[0015] The computer device proposed in the fifth aspect of the embodiment of the present invention obtains the image file generated by the node of the target type; parses the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merges the original file according to the information of the original file in combination with preset rules, thereby being able to automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0018] Figure 1 It is a flowchart of a file processing method proposed in one embodiment of the present invention;

[0019] Figure 2 A schematic diagram of an application scenario of an embodiment of the present invention;

[0020] Figure 3 is a structural schematic diagram of a file processing device proposed in one embodiment of the present invention;

[0021] Figure 4 is a structural schematic diagram of a file processing device proposed in another embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of the structure of a computer device proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. On the contrary, embodiments of the present invention include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0024] Figure 1 It is a flowchart of a file processing method proposed in one embodiment of the present invention.

[0025] This embodiment is described by taking the file processing method configured as a file processing device as an example.

[0026] In this embodiment, the file processing method can be configured in a file processing device, and the file processing device can be set in a server, or can also be set in an electronic device, which is not limited in this embodiment of the present application.

[0027] This embodiment takes the file processing method being configured in an electronic device as an example.

[0028] The file is a file in a data warehouse tool. The data warehouse tool includes a target type of node. The target type of node may be a NameNode node of a metadata management center.

[0029] It should be noted that the execution entity of the embodiment of the present application may be, for example, a central processing unit (CPU) in a server or electronic device in terms of hardware, or may be, for example, a related background service in a server or electronic device in terms of software, without limitation.

[0030] Data warehouse tools can map structured data files into a database table and provide simple SQL (Structured Query Language) functions, which can convert SQL statements into distributed computing tasks for execution.

[0031] Data warehouse tools generally run on the Hadoop distributed file system, and a large number of small files are generated during the operation. Small files may be generated when the data source is imported into the data warehouse tool, or when the data table of the data warehouse tool is read for offline calculation. Usually, a single file needs to occupy a calculation process or thread when calculating. A large number of small files consumes more computing resources. Therefore, it is necessary to process the files generated during the operation of the data warehouse tool.

[0032] In order to solve the above technical problems, a file processing method is provided in an embodiment of the present invention, which obtains the image file generated by the node of the target type; parses the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merges the original files according to the information of the original file in combination with preset rules, thereby realizing automatic identification of the files generated when the data warehouse tool is running, and merging the files in a timely manner.

[0033] See also Figure 1 , the method comprising:

[0034] S101: Acquire the image file generated by the node of the target type.

[0035] The target type of node may be a NameNode node of the metadata management center.

[0036] During the specific execution of the embodiment of the present invention, in order to realize automatic identification of files generated when the data warehouse tool is running, the image files generated by the target type of nodes can be stored in the local storage device during the operation of the data warehouse tool.

[0037] In the embodiment of the present invention, the image file generated by the NameNode node of the metadata management center can be directly obtained in the local storage device, and then the image file is parsed.

[0038] Among them, the image file generated by the NameNode node of the metadata management center specifically corresponds to the original file, and the original file is the file generated when the data warehouse tool is running.

[0039] The image file may be parsed in real time, or may be parsed at certain intervals, and there is no limitation to this.

[0040] S102: In combination with the directory information of the data warehouse tool, the image file is parsed to obtain information of the original file to which the image file belongs.

[0041] The original files are files corresponding to each database table and files corresponding to each partition in multiple database tables and multiple partitions of the data warehouse tool.

[0042] The directory information is used to describe the specific organizational structure of each database table and each partition.

[0043] The information includes: quantity and size of storage space occupied.

[0044] During the specific execution process of the embodiment of the present invention, the directory information of the data warehouse tool can be combined to determine the information of the first original file corresponding to each database table and the information of the second original file corresponding to each partition in multiple database tables and multiple partitions of the data warehouse tool.

[0045] The original file corresponding to the database table may be referred to as a first original file, and the original file corresponding to each partition may be referred to as a second original file.

[0046] By combining the directory information of the data warehouse tool, the information of the first original file corresponding to each database table and the information of the second original file corresponding to each partition in the multiple database tables and multiple partitions of the data warehouse tool are determined, so that the original files can be located in a timely and accurate manner, making it easier to obtain the information of the original files corresponding to the database tables in a timely manner.

[0047] In the embodiment of the present invention, the image file is parsed in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs, instead of directly accessing the NameNode node of the metadata management center through a preset interface. This can simplify the process of obtaining the information of the original file and improve the file processing efficiency. Moreover, since the NameNode node of the metadata management center is not directly accessed through a preset interface remote call, but the image file is obtained for analysis, no additional access pressure is brought to the NameNode node, and adverse effects on the stability of the production environment can be avoided.

[0048] S103: merging the original files according to the information of the original files and in combination with preset rules.

[0049] Optionally, in some embodiments, the original files are merged according to the information of the original files in combination with preset rules, including: determining a first average value of the size of the storage space occupied by the first original files corresponding to each database table based on the information of each first original file, and determining a second average value of the size of the storage space occupied by the second original files corresponding to each partition, and merging the original files according to the first average value, the second average value, the number of first original files, and the number of second original files in combination with preset rules.

[0050] Optionally, in some embodiments, the original files are merged based on the first average value, the second average value, the number of first original files, and the number of second original files in combination with preset rules, including: when the first average value or the second average value is less than or equal to a first preset threshold, the first original file or the second original file is merged; and / or when the number of the first original files or the number of the second original files is greater than the second preset threshold, the first original file or the second original file is merged.

[0051] The first preset threshold and the second preset threshold may be set by the user according to needs, or may be pre-set by a factory program of the electronic device, and there is no limitation on this.

[0052] By setting a first preset threshold and a second preset threshold, when the first average value or the second average value is less than or equal to the first preset threshold, the first original file or the second original file is merged; and / or when the number of the first original files or the number of the second original files is greater than the second preset threshold, the first original file or the second original file is merged. Reasonable judgment conditions for merging are set, which can not only realize automatic identification of files generated when the data warehouse tool is running, but also merge files in time. Moreover, since the first preset threshold and the second preset threshold can be set by the user according to needs, or can also be pre-set by the factory program of the electronic device, the flexibility of setting the merging timing is effectively improved, and the applicability of the method is improved.

[0053] In this embodiment, the image file generated by the node of the target type is obtained; the image file is parsed in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; the original files are merged according to the information of the original files in combination with preset rules, so that it is possible to automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.

[0054] As an example, see Figure 2 , Figure 2The following is a schematic diagram of an application scenario of an embodiment of the present invention. The NameNode node of the metadata management center will regularly record metadata information including file information on a local disk (the disk is the local storage device in the present invention), and the stored content can be called an image file (Image); the Analysis process is started regularly, and the image file generated by the NameNode node of the metadata management center is obtained in the background, and the image file is parsed, and the information of the original file to which the image file belongs is obtained according to the Hive directory (corresponding to a single table or a single partition of Hive), and the information can be, for example, the number of files and the size of the file; the parsed result is stored in any storage system that can be queried; the Hive directory information obtained by querying is combined with preset rules to calculate the Hive table or partition that needs to be merged, and the preset rules include: 1) the average value of the size of the storage space occupied by the original file is less than or equal to the first preset threshold; 2) the number of original files is greater than or equal to the second preset threshold. If all conditions are met, it is considered that there are too many small files in the corresponding directory and they need to be merged.

[0055] During the specific execution of the embodiment of the present invention, considering that the data itself is a business-sensitive resource, this process can be completed automatically by the electronic device, or it can be confirmed by relevant personnel, and instructions can be issued to merge the corresponding Hive tables or partitions, which can be specifically calculated through MapReduce / Spark or Hive.

[0056] Figure 3 It is a structural schematic diagram of a file processing device proposed in one embodiment of the present invention.

[0057] The file is a file in a data warehouse tool, and the data warehouse tool includes a node of a target type.

[0058] See also Figure 3 , the device 300 comprises:

[0059] An acquisition module 301 is used to acquire an image file generated by a node of a target type;

[0060] The parsing module 302 is used to parse the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs;

[0061] The merging processing module 303 is used to merge the original files according to the information of the original files and in combination with preset rules.

[0062] Optionally, in some embodiments, see Figure 4 , the device 300 further includes:

[0063] The storage module 304 is used to store the image files generated by the target type of nodes in the local storage device during the operation of the data warehouse tool.

[0064] Optionally, in some embodiments, the information includes: quantity and size of the storage space occupied, and the parsing module 302 is specifically used to:

[0065] In combination with the directory information of the data warehouse tool, information of the first original file corresponding to each database table and information of the second original file corresponding to each partition in the multiple database tables and multiple partitions of the data warehouse tool are determined.

[0066] Optionally, in some embodiments, the merging processing module 303 is specifically configured to:

[0067] Determine, based on information of each first original file, a first average value of the size of the storage space occupied by the first original file corresponding to each database table, and determine a second average value of the size of the storage space occupied by the second original file corresponding to each partition;

[0068] The original files are merged according to the first average value, the second average value, the number of the first original files, and the number of the second original files in combination with a preset rule.

[0069] Optionally, in some embodiments, the merging processing module 303 is specifically configured to:

[0070] When the first average value or the second average value is less than or equal to the first preset threshold, merging the first original file or the second original file; and / or,

[0071] When the number of the first original files or the number of the second original files is greater than a second preset threshold, the first original files or the second original files are merged.

[0072] It should be noted that the above Figure 1-Figure 2 The explanation of the file processing method embodiment in the embodiment is also applicable to the file processing device 300 of this embodiment. The implementation principle is similar and will not be repeated here.

[0073] In this embodiment, the image file generated by the node of the target type is obtained; the image file is parsed in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; the original files are merged according to the information of the original files in combination with preset rules, so that it is possible to automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.

[0074] Figure 5 It is a schematic diagram of the structure of a computer device proposed in one embodiment of the present invention.

[0075] The computer device may be a mobile phone, a tablet computer, etc.

[0076] See also Figure 5 The computer device 50 of this embodiment includes: a housing 501, a processor 502, a memory 503, a circuit board 504, and a power circuit 505. The circuit board 504 is arranged inside the space surrounded by the housing 501, and the processor 502 and the memory 503 are arranged on the circuit board 504; the power circuit 505 is used to supply power to various circuits or devices of the computer device 50; the memory 503 is used to store executable program codes; wherein the processor 502 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 503, so as to execute:

[0077] Get the image file generated by the node of the target type;

[0078] Combined with the directory information of the data warehouse tool, the image file is parsed to obtain the information of the original file to which the image file belongs;

[0079] The original files are merged based on the information of the original files and the preset rules.

[0080] It should be noted that the above Figure 1-Figure 2 The explanation of the file processing method in the embodiment is also applicable to the computer device 50 of this embodiment. The implementation principle is similar and will not be repeated here.

[0081] The computer device in this embodiment obtains the image file generated by the node of the target type; combines the directory information of the data warehouse tool to parse the image file to obtain the information of the original file to which the image file belongs; merges the original file according to the information of the original file and combines it with preset rules, so as to automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.

[0082] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute a file processing method, wherein the file is a file in a data warehouse tool, and the data warehouse tool includes a node of a target type. The method includes:

[0083] Get the image file generated by the node of the target type;

[0084] Combined with the directory information of the data warehouse tool, the image file is parsed to obtain the information of the original file to which the image file belongs;

[0085] The original files are merged based on the information of the original files and the preset rules.

[0086] The non-temporary computer-readable storage medium in this embodiment obtains the image file generated by the node of the target type; parses the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merges the original files according to the information of the original files in combination with preset rules, thereby automatically identifying the files generated when the data warehouse tool is running and merging the files in a timely manner.

[0087] In order to implement the above embodiment, the present invention further proposes a computer program product. When the instructions in the computer program product are executed by a processor, a file processing method is executed. The file is a file in a data warehouse tool. The data warehouse tool includes a node of a target type. The method includes:

[0088] Get the image file generated by the node of the target type;

[0089] Combined with the directory information of the data warehouse tool, the image file is parsed to obtain the information of the original file to which the image file belongs;

[0090] The original files are merged based on the information of the original files and the preset rules.

[0091] The computer program product in this embodiment obtains the image file generated by the node of the target type; parses the image file in combination with the directory information of the data warehouse tool to obtain the information of the original file to which the image file belongs; merges the original files according to the information of the original files in combination with preset rules, thereby automatically identifying the files generated when the data warehouse tool is running and merging the files in a timely manner.

[0092] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0093] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0094] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0095] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0096] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0097] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0099] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A file processing method, characterized in that: The file is a file in a data warehouse tool, the data warehouse tool includes a node of a target type, and the method includes the following steps: During the operation of the data warehouse tool, the image file generated by the node of the target type is stored in a local storage device; Acquire the image file generated by the node of the target type from the local storage device; Parsing the image file in combination with the directory information of the data warehouse tool to obtain information of the original file to which the image file belongs; The original files are merged based on the information of the original files and in combination with preset rules, so as to automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.

2. The file processing method according to claim 1, characterized in that: The information includes: the quantity and the size of the storage space occupied. The information of the original file to which the image file belongs is obtained by parsing the image file in combination with the directory information of the data warehouse tool, including: In combination with the directory information of the data warehouse tool, information of the first original file corresponding to each of the multiple database tables and multiple partitions of the data warehouse tool, and information of the second original file corresponding to each of the partitions are determined.

3. The file processing method according to claim 2, characterized in that: The merging process of the original files according to the information of the original files and in combination with preset rules includes: Determine, according to the information of each of the first original files, a first average value of the size of the storage space occupied by the first original files corresponding to each of the database tables, and determine a second average value of the size of the storage space occupied by the second original files corresponding to each of the partitions; The original files are merged according to the first average value, the second average value, the number of the first original files, and the number of the second original files in combination with a preset rule.

4. The file processing method according to claim 3, characterized in that: The merging process of the original files according to the first average value, the second average value, the number of the first original files, and the number of the second original files in combination with a preset rule includes: When the first average value or the second average value is less than or equal to a first preset threshold, merging the first original file or the second original file; and / or, When the number of the first original files or the number of the second original files is greater than a second preset threshold, the first original files or the second original files are merged.

5. A file processing device, characterized in that: The file is a file in a data warehouse tool, the data warehouse tool includes a node of a target type, and the device includes: A storage module, used to store the image files generated by the nodes of the target type in a local storage device during the operation of the data warehouse tool; An acquisition module, used for acquiring the image file generated by the node of the target type from the local storage device; A parsing module, used to parse the image file in combination with the directory information of the data warehouse tool to obtain information of the original file to which the image file belongs; The merging processing module is used to merge the original files according to the information of the original files and in combination with preset rules, so as to automatically identify the files generated when the data warehouse tool is running and merge the files in a timely manner.

6. The file processing device according to claim 5, characterized in that: The information includes: the quantity and the size of the storage space occupied. The parsing module is specifically used to: In combination with the directory information of the data warehouse tool, information of the first original file corresponding to each of the multiple database tables and multiple partitions of the data warehouse tool, and information of the second original file corresponding to each of the partitions are determined.

7. The file processing device according to claim 6, characterized in that: The merging processing module is specifically used for: Determine, according to the information of each of the first original files, a first average value of the size of the storage space occupied by the first original files corresponding to each of the database tables, and determine a second average value of the size of the storage space occupied by the second original files corresponding to each of the partitions; The original files are merged according to the first average value, the second average value, the number of the first original files, and the number of the second original files in combination with a preset rule.

8. The file processing device according to claim 7, characterized in that: The merging processing module is specifically used for: When the first average value or the second average value is less than or equal to a first preset threshold, merging the first original file or the second original file; and / or, When the number of the first original files or the number of the second original files is greater than a second preset threshold, the first original files or the second original files are merged.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the file processing method according to any one of claims 1 to 4 is implemented.

10. A computer device comprising a housing, a processor, a memory, a circuit board and a power supply circuit, wherein: The circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the computer device; the memory is used to store executable program codes; the processor runs a program corresponding to the executable program codes by reading the executable program codes stored in the memory, so as to execute: During the operation of the data warehouse tool, the image files generated by the target type of nodes are stored in the local storage device; Acquire the image file generated by the node of the target type from the local storage device; Parsing the image file in combination with the directory information of the data warehouse tool to obtain information of the original file to which the image file belongs; The original files are merged based on the information of the original files and in combination with preset rules, so as to automatically identify the files generated when the data warehouse tool is running, and merge the files in a timely manner.