File processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a file processing method and device, electronic equipment and a storage medium. The method comprises the following steps: in response to a Spark task processing request, obtaining a plurality of small files corresponding to the Spark task processing request, and storing each small file into a folder corresponding to each small file; the method comprises the following steps: for each folder, under the condition that the folder comprises a first file, determining a plurality of to-be-merged files in the folder based on a preset file merging condition, and sequentially writing file contents of the plurality of to-be-merged files into the first file according to a file merging sequence of the to-be-merged files, and storing the first file into a target data storage cluster. According to the technical scheme, the small files are merged in parallel in the Spark data processing process, so that the number of the small files is reduced, and the data storage pressure of a data storage cluster is reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer processing technology, and in particular to a file processing method, device, electronic device and storage medium. Background Art

[0002] Spark is a powerful distributed computing framework for high-performance computing of large-scale data sets. It is widely used in various industries such as finance, telecommunications, medical care, retail, media, etc. to process and analyze big data. Currently, when using the Spark engine for big data processing, especially in real-time processing or high-frequency batch processing tasks, a large number of small files are usually generated.

[0003] In the related art, a large number of small files generated by Spark engine processing are usually directly stored in the data storage cluster, which greatly increases the storage pressure of the data storage cluster. Therefore, a file processing method is proposed to parallel merge the small files generated by Spark task processing to reduce the number of small files and reduce the data storage pressure of the data storage cluster. Summary of the invention

[0004] The present invention provides a file processing method, device, electronic device and storage medium to realize parallel merging of small files during Spark task processing, so as to reduce the number of small files and reduce the data storage pressure of a data storage cluster.

[0005] According to one aspect of the present invention, there is provided a file processing method, the method comprising:

[0006] In response to a Spark task processing request, a plurality of small files corresponding to the Spark task processing request are obtained, and each of the small files is stored in a folder corresponding to each small file;

[0007] For each folder, when the folder includes a first file, based on a preset file merging condition, multiple files to be merged in the folder are determined, and according to the file merging order of the files to be merged, the file contents of the multiple files to be merged are written into the first file in sequence to store the first file in the target data storage cluster.

[0008] Optionally, in response to a Spark task processing request, multiple small files corresponding to the Spark task processing request are obtained, and each of the small files is stored in a folder corresponding to each small file, including: in response to a Spark task processing request, multiple task partitions of the Spark task are determined, the data in each task partition is processed respectively to obtain small files corresponding to each task partition, and the small files corresponding to each task partition are stored in the folder corresponding to each task partition.

[0009] Optionally, before storing the small files corresponding to each task partition in the folder corresponding to each task partition, the method also includes: for each task partition, creating a folder corresponding to the task partition to store the files obtained after processing the data in the task partition; wherein the file identifier of the folder includes the Spark task identifier.

[0010] Optionally, after writing the file contents of multiple files to be merged into the first file in sequence, the method further includes: obtaining a file merge correspondence relationship, and recording the file merge correspondence relationship in a preset index file of the folder; wherein the file merge correspondence relationship describes the correspondence between the first file identifiers of multiple files to be merged and the second file identifier of the first file.

[0011] Optionally, after writing the file contents of multiple files to be merged into the first file in sequence, the method further includes: when the file format of the first file is not the target format, creating a second file in the target format in the folder, writing the file content of the first file into the second file, and replacing the first file in the target data storage cluster with the second file.

[0012] Optionally, the preset file merging condition includes at least one of the following conditions: the number of small files in the folder exceeds a preset file number threshold; the file size of the small files in the folder exceeds a preset file size threshold; the file generation time of the small files in the folder reaches a preset deletion time; the current merge time of the small files in the folder reaches a preset merge time from the previous merge time.

[0013] Optionally, after writing the file contents of the plurality of files to be merged into the first file in sequence, the method further includes: deleting the plurality of files to be merged in the folder.

[0014] According to another aspect of the present invention, a file processing device is provided. The device comprises:

[0015] A file storage module is used to respond to a Spark task processing request, obtain multiple small files corresponding to the Spark task processing request, and store each of the small files in a folder corresponding to each small file;

[0016] A file merging module is used to determine, for each folder, a plurality of files to be merged in the folder based on a preset file merging condition when the folder includes a first file, and write the file contents of the plurality of files to be merged into the first file in sequence according to a file merging order of the files to be merged, so as to store the first file in a target data storage cluster.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file processing method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file processing method described in any embodiment of the present invention when executed.

[0022] The technical solution of the embodiment of the present invention obtains multiple small files corresponding to the Spark task processing request by responding to the Spark task processing request, and stores each of the small files in a folder corresponding to each small file; for each folder, when the folder includes a first file, based on a preset file merging condition, multiple files to be merged in the folder are determined, and according to the file merging order of the files to be merged, the file contents of the multiple files to be merged are sequentially written into the first file, so as to store the first file in the target data storage cluster. The technical solution of the embodiment of the present invention realizes the parallel merging of small files in the Spark data processing process, so as to reduce the number of small files and reduce the data storage pressure of the data storage cluster.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a file processing method provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of a file processing method provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the structure of a file processing device provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0032] Figure 1The present invention provides a flowchart of a file processing method. This embodiment is applicable to file processing. The method can be executed by a file processing device. The file processing device can be implemented in the form of hardware and / or software. The file processing device can be configured in an electronic device such as a computer or a server. Figure 1 As shown, the method of this embodiment includes:

[0033] S110. In response to a Spark task processing request, a plurality of small files corresponding to the Spark task processing request are obtained, and each of the small files is stored in a folder corresponding to each small file.

[0034] In the embodiment of the present invention, the Spark task processing request can be understood as a request for triggering the Spark data processing engine to perform task processing on the task to be processed. The task to be processed may include a task for performing data processing on the data to be processed. The data to be processed may include at least one of image data, text data, audio data, etc. Optionally, the task to be processed may include at least one data processing task of data screening, data aggregation, data classification, etc.

[0035] In an embodiment of the present invention, a method of obtaining a Spark task processing request may specifically be that, when a Spark task is received, a task processing request for processing the Spark task, i.e., a Spark task processing request, may be generated. In an embodiment of the present invention, a large number of small files may be generated during the execution of a Spark task. In an embodiment of the present invention, there are multiple ways to obtain a Spark task. For example, it may be a pre-set data processing task for processing data to be processed; or, it may be a data processing task sent by a receiving client for processing data to be processed.

[0036] As an optional implementation in the embodiment of the present invention, a plurality of small files corresponding to the Spark task processing request are obtained, and each of the small files is stored in a folder corresponding to each small file, including: during the task processing process of the Spark task processing request, when one or more small files are detected to be generated, each generated small file can be stored in a corresponding folder. This technical solution realizes the real-time storage of the small files generated during the task processing process, and ensures the immediacy of file processing.

[0037] On the basis of the above embodiment, for each small file generated during the task processing of the Spark task processing request, it can be determined whether there is a folder for storing the small file. If so, the small file can be stored in the folder for storing the small file. If not, a folder for storing the small file can be created, and the small file is stored in the created folder for storing the small file.

[0038] As another optional implementation in the embodiment of the present invention, multiple small files corresponding to the Spark task processing request are obtained, and each of the small files is stored in a folder corresponding to each small file, including: after the task processing of the Spark task processing request is completed, all small files generated after the Spark task processing request is processed can be obtained. Then, each small file can be stored in a folder corresponding to each small file. In this technical solution, all small files generated by task processing are uniformly stored after task processing is completed.

[0039] On the basis of the above embodiment, after obtaining all the small files generated after the Spark task processing request is processed, for each small file, it can be determined whether there is a folder for storing the small file. If so, the small file can be stored in the folder. If not, a folder for storing the small file can be created and the small file can be stored in the folder.

[0040] S120. For each folder, when the folder includes a first file, based on a preset file merging condition, determine a plurality of files to be merged in the folder, and write the file contents of the plurality of files to be merged into the first file in sequence according to a file merging order of the files to be merged, so as to store the first file in a target data storage cluster.

[0041] Among them, the first file can be understood as a file whose file content is to be written into the merged file. The preset file merge condition can be understood as a pre-set condition for merging multiple small files generated in a folder. In an embodiment of the present invention, the preset file merge condition includes at least one of the following conditions: the number of small files in the folder exceeds the preset file number threshold; the file size of the small files in the folder exceeds the preset file size threshold; the file generation time of the small files in the folder reaches the preset deletion time; the current merge time of the small files in the folder reaches the preset merge time from the previous merge time. In an embodiment of the present invention, the preset file number threshold, the preset file size threshold, the preset deletion time, and the preset merge time can be set according to actual needs, and their values ​​are not specifically limited here.

[0042] In an embodiment of the present invention, the files to be merged can be understood as small files that need to be merged in a folder. The file merging order can be understood as the merging order of the files to be merged, which is used to determine the writing order of the files to be merged in the first file. In an embodiment of the present invention, the file merging order can be obtained based on the generation time of the files to be merged, or it can be obtained based on the file size of the files to be merged, or it can be randomly generated for multiple files to be merged. The target data storage dataset can be understood as a data storage dataset for storing the files generated by the Spark task. For example, a Hadoop data storage dataset. It can be understood that when the target data storage dataset is a Hadoop data storage dataset, the small files generated by the Spark task processing can be small files in Hive, a data warehouse tool based on Hadoop.

[0043] As an optional implementation in an embodiment of the present invention, in the process of processing a Spark task, after each of the small files is stored in a folder corresponding to each small file, or when each of the small files is stored in a folder corresponding to each small file, for each folder, when the folder includes a first file, a plurality of files to be merged in the folder can be determined based on a preset file merge condition. Thus, the file contents of the plurality of files to be merged can be sequentially written into the first file according to the file merge order of the files to be merged, so that the first file is stored in the target data storage cluster. This technical solution realizes parallel file merge processing and improves the processing efficiency of file merge.

[0044] As another implementation in the embodiment of the present invention, after the Spark task processing is completed and each of the small files is stored in a folder corresponding to each small file, for each folder, when the folder includes the first file, multiple files to be merged in the folder can be determined based on a preset file merging condition. Thus, the file contents of multiple files to be merged can be sequentially written into the first file according to the file merging order of the files to be merged, so that the first file is stored in the target data storage cluster.

[0045] It can be understood that after each of the small files is stored in a folder corresponding to each small file, for each folder, if the first file does not exist in the folder, a file for writing the file content of the file to be merged can be created, that is, a first file is created to write the file content of the file to be merged.

[0046] In an embodiment of the present invention, after determining the files to be merged in the folder, writing the file contents of the multiple files to be merged into the first file in sequence according to the file merging order of the files to be merged includes: determining the current file to be merged among the multiple files to be merged according to the file merging order of the files to be merged, writing the file contents of the current file to be merged into the first file, and updating the current file to be merged among the multiple files to be merged based on the file merging order of the files to be merged, and re-performing the step of writing the file contents of the current file to be merged into the first file until all the files to be merged are written into the first file. Thus, writing the file contents of the multiple files to be merged into the first file in sequence is achieved.

[0047] In an embodiment of the present invention, after writing the file contents of multiple files to be merged into the first file in sequence, the method may further include: obtaining a file merge correspondence relationship, and recording the file merge correspondence relationship in a preset index file of the folder; wherein the file merge correspondence relationship describes the correspondence between the first file identifiers of the multiple files to be merged and the second file identifier of the first file.

[0048] In an embodiment of the present invention, the preset index file also records the correspondence between the third file identifier of the unmerged small file and the second file identifier of the first file, as well as the file generation time of each small file and the file merging information. Among them, the file merging information can identify whether the small file has been merged. The file merging information may include merged and unmerged. In the case where the file merging information of the small file is merged, it can be characterized that the small file has been merged. In the case where the file merging information of the small file is unmerged, it can be characterized that the small file is unmerged. The first file identifier can be understood as the file identifier of the file to be merged. The second file identifier can be understood as the file identifier of the first file. The file identifier of the third file can be understood as the file identifier of the unmerged small file.

[0049] On the basis of the above embodiment, after the file contents of multiple files to be merged are written into the first file in sequence, the method may further include: when the file format of the first file is not the target format, a second file of the target format may be created in the folder, the file contents of the first file may be written into the second file, and the first file in the target data storage cluster may be replaced with the second file. Among them, the target format may be set according to the requirements, which is not specifically limited here. For example, it may be a text format of .orc. The second file may be understood as a file of the target format. In an embodiment of the present invention, after the file contents of the first file are written into the second file, the first file may be deleted.

[0050] In an embodiment of the present invention, after sequentially writing the file contents of the plurality of files to be merged into the first file, the method may further include: deleting the plurality of files to be merged in the folder. The benefit of such processing is that data storage space can be greatly reduced.

[0051] The technical solution of the embodiment of the present invention, in response to a Spark task processing request, obtains multiple small files corresponding to the Spark task processing request, and stores each of the small files in a folder corresponding to each small file; for each folder, when the folder includes a first file, based on a preset file merging condition, determines multiple files to be merged in the folder, and writes the file contents of the multiple files to be merged into the first file in sequence according to the file merging order of the files to be merged, so as to store the first file in the target data storage cluster. The technical solution of the embodiment of the present invention realizes the parallel merging of small files in the Spark data processing process, so as to reduce the number of small files and reduce the data storage pressure of the data storage cluster.

[0052] Figure 2 A flowchart of a file processing method provided for an embodiment of the present invention, based on the aforementioned embodiment, optionally, in response to a Spark task processing request, a plurality of small files corresponding to the Spark task processing request are obtained, and each of the small files is stored in a folder corresponding to each small file, including: in response to a Spark task processing request, a plurality of task partitions of a Spark task are determined, the data in each task partition is processed respectively, a small file corresponding to each task partition is obtained, and the small file corresponding to each task partition is stored in a folder corresponding to each task partition. Among them, the technical features identical or similar to those in the aforementioned embodiment are not repeated here. Figure 2 As shown, the method of this embodiment specifically includes:

[0053] S210. In response to a Spark task processing request, determine multiple task partitions of the Spark task, process the data in each task partition respectively, obtain small files corresponding to each task partition, and store the small files corresponding to each task partition in a folder corresponding to each task partition.

[0054] As an optional implementation of an embodiment of the present invention, in response to a Spark task processing request, multiple task partitions of a Spark task are determined so that the data in each task partition can be processed separately. In the process of processing the data in each task partition, a small file corresponding to each task partition can be obtained, and the small file corresponding to each task partition is stored in a folder corresponding to each task partition.

[0055] As another optional implementation of the embodiment of the present invention, in response to a Spark task processing request, multiple task partitions of the Spark task are determined so that the data in each task partition can be processed separately. After the data in each task partition is processed, a small file corresponding to each task partition can be obtained, and the small file corresponding to each task partition is stored in a folder corresponding to each task partition.

[0056] In an embodiment of the present invention, before storing the small files corresponding to each task partition in the folder corresponding to each task partition, the method may further include: for each task partition, a folder corresponding to the task partition may be created to store files obtained after processing the data in the task partition. The file identifier of the folder may include the Spark task identifier.

[0057] S220. For each folder, when the folder includes a first file, based on a preset file merging condition, determine a plurality of files to be merged in the folder, and write the file contents of the plurality of files to be merged into the first file in sequence according to a file merging order of the files to be merged, so as to store the first file in a target data storage cluster.

[0058] The technical solution of the embodiment of the present invention determines multiple task partitions of the Spark task in response to a Spark task processing request, processes the data in each task partition respectively, obtains small files corresponding to each task partition, and stores the small files corresponding to each task partition in a folder corresponding to each task partition. The technical solution of the embodiment of the present invention sets separate folders for different task partitions to store small files generated in the corresponding task partitions.

[0059] Figure 3 A schematic diagram of the structure of a file processing device provided by an embodiment of the present invention. Figure 3 As shown, the device includes: a file storage module 310 and a file merging module 320.

[0060] Among them, the file storage module 310 is used to respond to a Spark task processing request, obtain multiple small files corresponding to the Spark task processing request, and store each of the small files in a folder corresponding to each small file; the file merging module 320 is used to, for each folder, determine multiple files to be merged in the folder based on preset file merging conditions when the folder includes a first file, and write the file contents of the multiple files to be merged into the first file in sequence according to the file merging order of the files to be merged, so as to store the first file in the target data storage cluster.

[0061] The technical solution of the embodiment of the present invention is to obtain multiple small files corresponding to the Spark task processing request through a file storage module in response to the Spark task processing request, and store each of the small files in a folder corresponding to each small file; for each folder, the file merging module determines multiple files to be merged in the folder based on preset file merging conditions when the folder includes a first file, and writes the file contents of the multiple files to be merged into the first file in sequence according to the file merging order of the files to be merged, so as to store the first file in the target data storage cluster. The technical solution of the embodiment of the present invention realizes the parallel merging of small files in the Spark data processing process to reduce the number of small files and reduce the data storage pressure of the data storage cluster.

[0062] Optionally, the file storage module 310 is used to respond to the Spark task processing request, determine multiple task partitions of the Spark task, process the data in each task partition respectively, obtain small files corresponding to each task partition, and store the small files corresponding to each task partition in a folder corresponding to each task partition.

[0063] Optionally, the device further includes a folder creation module. The folder creation module is used to create a folder corresponding to each task partition for each task partition before storing the small files corresponding to each task partition in the folder corresponding to each task partition, so as to store the files obtained after processing the data in the task partition; wherein the file identifier of the folder includes the Spark task identifier.

[0064] Optionally, the device further comprises a relationship recording module, wherein the relationship recording module is used to obtain a file merging correspondence relationship after sequentially writing the file contents of the plurality of files to be merged into the first file, and record the file merging correspondence relationship in a preset index file of the folder; wherein the file merging correspondence relationship describes the correspondence relationship between the first file identifiers of the plurality of files to be merged and the second file identifier of the first file.

[0065] Optionally, the device further includes file format processing, wherein the file format processing is used to, after sequentially writing the file contents of the plurality of files to be merged into the first file, create a second file in the target format in the folder if the file format of the first file is not the target format, write the file contents of the first file into the second file, and replace the first file in the target data storage cluster with the second file.

[0066] Optionally, the preset file merging condition includes at least one of the following conditions: the number of small files in the folder exceeds a preset file number threshold; the file size of the small files in the folder exceeds a preset file size threshold; the file generation time of the small files in the folder reaches a preset deletion time; the current merge time of the small files in the folder reaches a preset merge time from the previous merge time.

[0067] Optionally, the device further comprises a file deletion module, wherein the file deletion module is used to delete the plurality of files to be merged in the folder after the file contents of the plurality of files to be merged are sequentially written into the first file.

[0068] The file processing device provided in the embodiment of the present invention can execute the file processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0069] It is worth noting that the various units and modules included in the above-mentioned file processing device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0070] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0071] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0072] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0073] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a file processing method.

[0074] In some embodiments, the file processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the file processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the file processing method in any other appropriate manner (e.g., by means of firmware).

[0075] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0076] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0077] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0078] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0079] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0080] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0081] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0082] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A file processing method, characterized in that: include: In response to a Spark task processing request, a plurality of small files corresponding to the Spark task processing request are obtained, and each of the small files is stored in a folder corresponding to each small file; For each folder, when the folder includes a first file, based on a preset file merging condition, multiple files to be merged in the folder are determined, and according to the file merging order of the files to be merged, the file contents of the multiple files to be merged are written into the first file in sequence to store the first file in the target data storage cluster.

2. The method according to claim 1, characterized in that: The step of obtaining, in response to the Spark task processing request, a plurality of small files corresponding to the Spark task processing request, and storing each of the small files in a folder corresponding to each small file comprises: In response to a Spark task processing request, multiple task partitions of the Spark task are determined, the data in each task partition is processed respectively to obtain small files corresponding to each task partition, and the small files corresponding to each task partition are stored in a folder corresponding to each task partition.

3. The method according to claim 2, characterized in that Before storing the small files corresponding to each task partition in the folder corresponding to each task partition, the method further includes: For each task partition, a folder corresponding to the task partition is created to store files obtained after processing the data in the task partition; wherein the file identifier of the folder includes the Spark task identifier.

4. The method according to claim 1, characterized in that After sequentially writing the file contents of the plurality of files to be merged into the first file, the method further includes: Obtaining a file merging correspondence relationship, and recording the file merging correspondence relationship in a preset index file of the folder; The file merging correspondence relationship describes the correspondence between the first file identifiers of the plurality of files to be merged and the second file identifiers of the first files.

5. The method according to claim 1, characterized in that After sequentially writing the file contents of the plurality of files to be merged into the first file, the method further includes: When the file format of the first file is not the target format, a second file of the target format is created in the folder, the file content of the first file is written to the second file, and the first file in the target data storage cluster is replaced by the second file.

6. The method according to claim 1, characterized in that The preset file merging condition includes at least one of the following conditions: The number of small files in the folder exceeds a preset file number threshold; The file size of the small file in the folder exceeds a preset file size threshold; The file generation time of the small files in the folder reaches the preset deletion time; The current merge time of the small files in the folder reaches the preset merge time from the last merge time.

7. The method according to claim 1, characterized in that After sequentially writing the file contents of the plurality of files to be merged into the first file, the method further includes: Delete a plurality of the files to be merged in the folder.

8. A file processing device, characterized in that: include: A file storage module is used to respond to a Spark task processing request, obtain multiple small files corresponding to the Spark task processing request, and store each of the small files in a folder corresponding to each small file; A file merging module is used to determine, for each folder, a plurality of files to be merged in the folder based on a preset file merging condition when the folder includes a first file, and write the file contents of the plurality of files to be merged into the first file in sequence according to a file merging order of the files to be merged, so as to store the first file in a target data storage cluster.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the file processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the file processing method according to any one of claims 1 to 7 when executed.

Images