Method, system, and storage medium for file splitting

Abstract

The application relates to a file splitting method, system and storage medium, which comprises the following steps: acquiring key data of each line of data in a source file to be split; and splitting the source file by acquiring a corresponding file output stream object from a Map container according to the key data of each line of data in the source file, wherein the Map container is a file output stream object pool used for storing file output stream objects, and the splitting efficiency of the source file is improved without frequently creating and closing the file output stream objects.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to methods, systems and storage media for file splitting. Background Technology

[0002] When a computer is running, due to limitations in storage media capacity and network transmission capabilities, it is necessary to split large source files, that is, to save the data in the source file into multiple smaller split files for easier storage or transmission. In related technologies, the source file can be split by reading line by line and continuously creating and closing file output streams. However, frequent creation and closing of file output streams leads to poor performance and low efficiency in splitting the source file.

[0003] Currently, no effective solution has been proposed to address the problem of low efficiency in splitting source files due to the frequent creation and closure of file output streams required for splitting source files in related technologies. Summary of the Invention

[0004] This application provides a method, system, and storage medium for splitting files, which at least solves the problem in related technologies where splitting source files requires frequent creation and closure of file output streams, resulting in low efficiency in splitting source files.

[0005] In a first aspect, embodiments of this application provide a method for file splitting, the method comprising:

[0006] Obtain the key data for each line of data in the source file to be split;

[0007] Based on the key data of each line of data in the source file, the corresponding file output stream object is obtained from the Map container to split the source file until the source file is completely split. The Map container is a file output stream object pool used to store file output stream objects.

[0008] In some embodiments, before obtaining the key data for each line of data in the source file to be split, the method further includes:

[0009] Create a Map container;

[0010] Obtain the key data of each line of data in the source file to be split, create an equal number of file output stream objects according to the number of key data types, and save the file output stream objects and the key data into the Map container according to the mapping relationship.

[0011] In some embodiments, creating an equal number of file output stream objects based on the number of key data types, and saving the file output stream objects and the key data into the Map container according to the mapping relationship includes:

[0012] Read the key data of the current line of data in the source file and determine whether the key data exists in the Map container;

[0013] If the judgment result is negative, a file output stream object is created, and the newly created file output stream object and the corresponding key data are saved into the Map container according to the mapping relationship.

[0014] If the judgment result is yes, then obtain the key data of the next line of data in the source file, and determine whether the key data exists in the Map container. Repeat the previous execution step according to the judgment result until the source file is completely read.

[0015] In some embodiments, splitting the source file by retrieving the corresponding file output stream object from the Map container based on the key data of each line of data in the source file includes:

[0016] Read the key data of the current line of data in the source file and determine whether the key data exists in the Map container;

[0017] If the judgment result is yes, then obtain the file output stream object corresponding to the key data, and write the current line data into the corresponding split file through the file output stream object;

[0018] If the judgment result is negative, a file output stream object is created, the current line data is written to the corresponding split file through the newly created file output stream object, and the newly created file output stream object and the corresponding key data are saved to the Map container according to the mapping relationship.

[0019] Obtain the key data of the next line of data in the source file, and determine whether the key data exists in the Map container. Repeat the previous execution step according to the determination result until the source file is completely split.

[0020] In some embodiments, before saving the newly created file output stream object and its corresponding key data to the Map container according to the mapping relationship, the method further includes:

[0021] If the number of file output stream objects in the Map container reaches a preset threshold, close the newly created file output stream object or update the file output stream objects in the Map container.

[0022] In some embodiments, when key data of the line data in the source file cannot be obtained, the method further includes:

[0023] The line data for which no key data can be obtained is recorded as the target line data, and a target file output stream object is created for the target line data.

[0024] In some embodiments, after splitting the source file, the method further includes: traversing the Map container and closing all file output stream objects in the Map container.

[0025] In some embodiments, key data for obtaining each line of data in the source file includes:

[0026] Based on the format of the source file, a regular expression is constructed, and the key data in each line of data is obtained through the regular expression.

[0027] Secondly, embodiments of this application provide a file splitting system, the system including an acquisition module and a splitting module.

[0028] The acquisition module is used to acquire key data for each line of data in the source file to be split.

[0029] The splitting module is used to split the source file by obtaining the corresponding file output stream object from the Map container based on the key data of each line of data in the source file. The Map container is a file output stream object pool used to store file output stream objects.

[0030] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the file splitting method as described in the first aspect above.

[0031] Compared to related technologies, the file splitting method provided in this application obtains key data for each line of data in the source file to be split; based on the key data for each line of data in the source file, the corresponding file output stream object is obtained from the Map container to split the source file. The Map container is a file output stream object pool used to store file output stream objects. When splitting the source file, there is no need to frequently create and close file output stream objects, which improves the efficiency of source file splitting. Attached Figure Description

[0032] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0033] Figure 1 This is a flowchart of a file splitting method according to an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of a file splitting method according to an embodiment of this application;

[0035] Figure 3This is a schematic diagram of another file splitting method according to an embodiment of this application;

[0036] Figure 4 This is a structural block diagram of a file splitting system according to an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0038] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0039] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0040] This embodiment provides a method for file splitting. Figure 1 This is a flowchart of a file splitting method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0041] Step S101: Obtain key data for each line of data in the source file to be split. In this embodiment, a file input stream is created and the source file is read line by line through a BufferReader to obtain key data for each line of data. The key data can be a unique identifier in each line of data, such as a user ID. The source file is split according to the user ID. For example, if the source file contains 10 user IDs, the data in the source file is split into 10 split files according to the user IDs.

[0042] Step S102: Based on the key data of each line of data in the source file, the corresponding file output stream object is obtained from the Map container to split the source file. The Map container is a pool of file output stream objects used to store these objects. In this embodiment, the corresponding file output stream object can be pre-created based on the key data of each line of data in the source file and stored in the Map container. Then, based on the key data of each line of data in the source file, the corresponding file output stream object is obtained from the Map container to split the source file.

[0043] In related technologies, since the same key data is not continuous in each line of the source file, and a file output stream object can only write data to the same split file, the source file needs to be split frequently by creating and closing file output stream objects, resulting in low efficiency in splitting the source file.

[0044] For example, if the first and second lines of the source file contain data for user A, the third line contains data for user B, and the fourth line contains data for user A, then after creating the first file output stream object to write the first and second lines of data to the split file A, the first file output stream object needs to be closed. Then, a second file output stream object needs to be created to write the third line of data to the split file B, and then the second file output stream object needs to be closed. Finally, a third file output stream object needs to be created to write the fourth line of data to the split file A, and then the third file output stream object needs to be closed. Since the source file contains a large number of user IDs, and the order of the same user IDs in each line of data is almost completely shuffled, it is necessary to frequently create and close file output stream objects to split the source file completely.

[0045] In the technical solution of this application, through steps S101 to S102, a corresponding file output stream object is pre-created based on the key data of each line of data in the source file and stored in a Map container. When splitting the source file, the key data of each line of data in the source file to be split is obtained, the corresponding file output stream object is obtained from the Map container, and the current line of data is written to the corresponding split file according to the file output stream object until the source file is split. There is no need to frequently create and close file output stream objects, which improves the efficiency of source file splitting.

[0046] For example, if the first and second lines of the source file contain data for user A, the third line contains data for user B, and the fourth line contains data for user A, then a file output stream object A is created for user A, and a file output stream object B is created for user B. These are stored in a Map container. When splitting the source file, if the key data for the first line is user A, then the file output stream object A is retrieved from the Map container and the first line of data is written to the corresponding split file A. If the key data for the second line is also user A, then the second line of data is written to the split file A using file output stream object A. If the key data for the third line is user B, then file output stream object A is saved back to the Map container, and file output stream object B is retrieved from the Map container and the third line of data is written to the corresponding split file B. If the key data for the fourth line is user A, then file output stream object A is retrieved from the Map container and the fourth line of data is written to the split file A, and so on, until the source file is completely split.

[0047] In some embodiments, a Map container is created before obtaining the key data for each line of data in the source file to be split;

[0048] Obtain the key data of each line of data in the source file to be split, create an equal number of file output stream objects according to the number of key data types, and save the file output stream objects and key data into a Map container according to the mapping relationship; where the key is the key data and the value is the file output stream object, the file output stream objects and key data are saved into the Map container in a key-value format.

[0049] Optionally, the mapping relationship between the file output stream objects stored in the Map container and the key data can be one-to-one or many-to-one. For example, when the key data is a user ID, and the first line of the source file contains data for user A, the second line for user B, and the third line for user C, if we want to split the data of users A and B into the same first split file, and the data of user C into another second split file, then the user IDs of users A and B are key data of one type, and the user ID of user C is key data of another type. We create a first file output stream object for users A and B, and a second file for user C. The source file is split into two file output stream objects, which are stored in a Map container. When splitting the source file, if the key data of the first line is user A, then the first file output stream object is retrieved from the Map container and the first line of data is written to the corresponding first split file. If the key data of the second line is user B, then the first line of data is written to the corresponding first split file through the first file output stream object. If the key data of the third line is user C, then the first file output stream object is saved back to the Map container, and the second file output stream object is retrieved from the Map container and the third line of data is written to the corresponding second split file, until the source file is completely split.

[0050] In this embodiment, the number of key data types is the number of files to be split, which is also the number of file output stream objects required. For example, if the key data is a user ID and the source file contains data for user A and user B, then the number of key data types is 2. Therefore, at least 2 file output stream objects should be created to ensure that there are enough file output stream objects to split the source file.

[0051] In some embodiments, creating an equal number of file output stream objects based on the number of key data types, and saving the file output stream objects and key data to a Map container according to a mapping relationship includes:

[0052] The process reads the key data from the current line of the source file and checks if this key data exists in the Map container. If the check fails, a file output stream object is created, and the newly created file output stream object and its corresponding key data are saved to the Map container according to the mapping relationship. If the check succeeds, the key data from the next line of the source file is retrieved, and its existence in the Map container is checked. This process is repeated until the entire source file has been read. This embodiment creates the necessary file output stream objects for the key data in the source file and saves the file output stream objects and their corresponding key data to the Map container according to the mapping relationship.

[0053] In some of these embodiments, Figure 2 This is a schematic diagram of a file splitting method according to an embodiment of this application, such as... Figure 2 As shown, splitting the source file by retrieving the corresponding file output stream object from the Map container based on the key data of each line of data in the source file involves the following steps:

[0054] Step S201: Read the key data of the current line of data in the source file;

[0055] Step S202: Determine whether the key data exists in the Map container. If yes, proceed to step S203; otherwise, proceed to step S204.

[0056] Step S203: Obtain the file output stream object corresponding to the key data, and write the current line data to the corresponding split file through the file output stream object;

[0057] Step S204: Create a file output stream object, write the current line data to the corresponding split file through the newly created file output stream object, and save the newly created file output stream object and the corresponding key data to the Map container according to the mapping relationship;

[0058] Step S205: Determine if there is a next row of data. If yes, proceed to step S206; otherwise, proceed to step S207.

[0059] Step S206: Obtain the key data for the next row of data;

[0060] Step S207, End, meaning the source file has been split.

[0061] Compared to related technologies, which require frequent creation and closing of file output stream objects when splitting source files, resulting in low efficiency, the technical solution of this application, through steps S201 to S207, directly obtains the file output stream object corresponding to the key data when it is stored in the Map container, and writes the line data to the corresponding split file through the file output stream object. When the file output stream object corresponding to the key data is not stored in the Map container, it is created, and the newly created file output stream object and the corresponding key data are saved in the Map container according to the mapping relationship. This eliminates the need for frequent creation and closing of file output stream objects, thus improving the efficiency of source file splitting.

[0062] In some embodiments, before saving newly created file output stream objects and their corresponding key data to the Map container according to the mapping relationship, if the number of file output stream objects in the Map container reaches a preset threshold, the newly created file output stream objects are closed, or the file output stream objects in the Map container are updated. In this embodiment, the preset threshold can be set to the maximum number of file output stream objects that the Map container can store. Limiting the maximum number that the Map container can store can prevent memory overflow caused by too many file output stream objects stored in the Map container.

[0063] For example, if the maximum storage limit of the Map container is 50, and the source file contains data for 53 users, then when the 49th file output stream object is created, the number of file output stream objects has not reached the maximum storage limit of the Map container. When the 50th file output stream object is created, the number of file output stream objects reaches the maximum storage limit. In this case, the corresponding row data is written to the split file through the 50th file output stream object, and then the 50th file output stream object is closed. Alternatively, the file output stream objects in the Map container can be updated. For example, the least frequently used file output stream object in the Map container can be found, closed, and then the 50th file output stream object can be saved to the Map container. Or, for example, the first file output stream object saved to the Map container can be closed in a first-in-first-out manner, and then the 50th file output stream object can be saved to the Map container.

[0064] In related technologies, the efficiency of splitting the source file can be improved by reading the entire source file into memory at once and then splitting the source file by creating a unified file output stream. However, this method consumes too much memory and is prone to memory overflow. In this embodiment, the file output stream object is stored in a Map container and the maximum storage capacity of the Map container is limited, which can improve the efficiency of splitting the source file while preventing memory overflow.

[0065] In some embodiments, if key data for a line in the source file cannot be obtained, the line data for which key data cannot be obtained is designated as the target line data, and a target file output stream object is created for the target line data. In this embodiment, the target line data is uniformly split into a single split file. When the target line data is read, the target file output stream object is obtained from the Map container, and the target line data is written to the corresponding split file through the target file output stream object.

[0066] In some embodiments, after splitting the source file, the Map container is traversed, and all file output stream objects within the Map container are closed. In this embodiment, the Map container created for the source file is generally only applicable to that source file; therefore, after splitting the source file, all file output stream objects within the Map container are closed to reduce memory usage.

[0067] In some embodiments, obtaining key data for each line of data in the source file includes: constructing a regular expression based on the format of the source file, and using the regular expression to obtain the key data in each line of data. In this embodiment, based on the format of the source file, the position of the key data within the line of data can be determined, and a regular expression is constructed based on the position of the key data within the line of data to obtain the key data in each line of data.

[0068] In some of these embodiments, Figure 3 This is a schematic diagram illustrating another file splitting method according to an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps:

[0069] Step S301, the program begins;

[0070] Step S302: Create a file input stream, wherein the file input stream is used to read the source file;

[0071] Step S303: Create a Map container to store the file output stream object, where the key is the key data obtained from each line of data, such as userId, activityid, etc.

[0072] Step S304: Read the source file through the file input stream;

[0073] Step S305: Determine if there is a next line. If not, proceed to step S306; if yes, proceed to step S307.

[0074] Step S306, program ends;

[0075] Step S307: Obtain the key data of the row data;

[0076] Step S308: Determine whether the key data already exists in the Map container. If yes, proceed to step S309; ​​otherwise, proceed to step S310.

[0077] Step S309: Obtain the file output stream object corresponding to the key data key, and write the current row data into the split file through the file output stream object;

[0078] Step S310: Create the file output stream object corresponding to the key data key;

[0079] Step S311: Write the current row data into the split file using the newly created file output stream object;

[0080] Step S312: Determine whether the number of file output stream objects is greater than a preset number threshold. If yes, proceed to step S313; otherwise, proceed to step S314.

[0081] Step S313: Close the file output stream object and proceed to step S305;

[0082] Step S314: Store the key-value pair in the container Map, where the key is the key data and the value is the file output stream object, and then proceed to step S305.

[0083] By going through the above steps S301 to S314, when splitting source files, it is not necessary to frequently create and close file output stream objects, which improves the efficiency of splitting source files, and prevents memory overflow by limiting the maximum storage capacity of the Map container.

[0084] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0085] This embodiment also provides a file splitting system for implementing the above embodiments and preferred embodiments, which will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0086] Figure 4 This is a structural block diagram of a system for file splitting according to an embodiment of this application, such as... Figure 4As shown, the system includes an acquisition module 41 and a splitting module 42. The acquisition module 41 is used to acquire the key data of each line of data in the source file to be split. The splitting module 42 is used to split the source file by acquiring the corresponding file output stream object from the Map container based on the key data of each line of data in the source file. The Map container is a file output stream object pool used to store file output stream objects. When splitting the source file, there is no need to frequently create and close file output stream objects, which improves the splitting efficiency of the source file.

[0087] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0088] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0089] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0090] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0091] Furthermore, in conjunction with the file splitting methods described in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the file splitting methods described in the above embodiments.

[0092] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a file splitting method. The display screen of the computer device may be a liquid crystal display (LCD) or an e-ink display. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad located on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0093] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0094] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method of file splitting, characterized by, The method comprises: creating a Map container, obtaining key data of each line of data in a source file to be split, creating a same number of file output stream objects according to the number of key data types, and saving the file output stream objects and the key data in a mapping relationship to the Map container, in a case where the number of file output stream objects in the Map container reaches a preset number threshold, closing a newly created file output stream object, or updating the file output stream object in the Map container, wherein the preset number threshold is set as the maximum number of file output stream objects stored in the Map container, and is used to limit the maximum number of file output stream objects stored in the Map container; splitting the source file according to the key data of each line of data in the source file, until the source file is split completely, wherein the Map container is a file output stream object pool, and is used to store file output stream objects; the creating a same number of file output stream objects according to the number of key data types, and saving the file output stream objects and the key data in a mapping relationship to the Map container comprises: reading key data of current line data in a source file, and judging whether the key data exists in the Map container; if the judgment result is no, creating a file output stream object, and saving the newly created file output stream object and corresponding key data in a mapping relationship to the Map container; if the judgment result is yes, obtaining key data of next line data in the source file, and judging whether the key data exists in the Map container, according to the judgment result, repeating the previous execution step until the source file is read completely; the splitting the source file according to the key data of each line of data in the source file, and obtaining corresponding file output stream objects from the Map container comprises: reading key data of current line data in a source file, and judging whether the key data exists in the Map container; if the judgment result is yes, obtaining the file output stream object corresponding to the key data, and writing the current line data into a corresponding split file through the file output stream object; if the judgment result is no, creating a file output stream object, writing the current line data into a corresponding split file through the newly created file output stream object, and saving the newly created file output stream object and corresponding key data in a mapping relationship to the Map container; obtaining key data of next line data in the source file, and judging whether the key data exists in the Map container, according to the judgment result, repeating the previous execution step until the source file is split completely.

2. The method of claim 1, wherein, in a case where the key data of line data in the source file cannot be obtained, the method further comprises: recording the line data for which the key data cannot be obtained as target line data, and creating a target file output stream object for the target line data.

3. The method of claim 1, wherein, after the source file is split completely, the method further comprises: traversing the Map container, and closing all file output stream objects in the Map container.

4. The method of claim 1, wherein, the obtaining key data of each line of data in the source file comprises: According to the format of the source file, a regular expression is constructed, and key data in each line of data is obtained through the regular expression.

5. A system for file splitting, characterized by The system comprises an obtaining module and a splitting module, The obtaining module is configured to create a Map container, obtain key data of each line of data in a source file to be split, create a same number of file output stream objects according to the number of key data types, save the file output stream objects and the key data in a mapping relationship to the Map container, and close a newly created file output stream object or update a file output stream object in the Map container when the number of file output stream objects in the Map container reaches a preset number threshold, wherein the preset number threshold is set as the maximum number of file output stream objects stored in the Map container, and is used to limit the maximum number of file output stream objects stored in the Map container. The splitting module is configured to split the source file according to the key data of each line of data in the source file, by obtaining corresponding file output stream objects from the Map container, wherein the Map container is a file output stream object pool, and is used to store file output stream objects. The method comprises the following steps: reading key data of a current line of data in a source file, and judging whether the key data exists in the Map container; if the result of the judgment is no, creating a file output stream object, and saving the newly created file output stream object and corresponding key data in a mapping relationship to the Map container; if the result of the judgment is yes, obtaining key data of a next line of data in the source file, and judging whether the key data exists in the Map container, and repeating the previous execution step according to the result of the judgment, until the source file is read completely. The method comprises the following steps: reading key data of a current line of data in a source file, and judging whether the key data exists in the Map container; if the result of the judgment is yes, obtaining a file output stream object corresponding to the key data, and writing the current line of data into a corresponding split file through the file output stream object; if the result of the judgment is no, creating a file output stream object, writing the current line of data into a corresponding split file through the newly created file output stream object, and saving the newly created file output stream object and corresponding key data in a mapping relationship to the Map container; obtaining key data of a next line of data in the source file, and judging whether the key data exists in the Map container, and repeating the previous execution step according to the result of the judgment, until the source file is split completely.

6. A storage medium, characterized by The storage medium stores a computer program, wherein the computer program is configured to execute the method for splitting a file according to any one of claims 1 to 4 when running.

Images