A method and apparatus for splitting files

Abstract

This invention provides a file splitting method and apparatus suitable for distributed clusters. Multiple servers in the distributed cluster execute their respective splitting tasks. The method includes: for any one of the multiple servers, the server obtains a fragment to be split from the file to be split according to the fragment position in the splitting task; the server determines the category corresponding to each line of data in the fragment to be split according to the business logic in the file processing configuration and records it in the local variable of the category; the server writes the local variables of each category into the respective category file in a locked manner according to the category storage information in the file processing configuration. Using this method, the file splitting speed can be accelerated, reducing time and labor costs.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a method and apparatus for file splitting. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology. In recent years, the application of computer technology has led to rapid development in the financial industry. However, this rapid development has also resulted in an explosive increase in financial-related data, making it difficult for existing computer technologies to support the industry's continued growth.

[0003] In existing technologies, by setting up business logic and related logic to run that business logic on a single server, the required data can be obtained by running that business logic. For example, if a user needs to split a client financial information file into Shanghai, Beijing, and Guangzhou clients, the business logic for splitting the file into these categories and the related logic can be written on a single server. By running this business logic, the file can be split into the required categories. Compared to traditional manual splitting, this saves a significant amount of manpower and resources. However, with the popularization of financial management concepts and the increase in the number of people investing, this method can no longer meet user needs.

[0004] Therefore, there is an urgent need for a file splitting method and device that can speed up file splitting and reduce time and labor costs. Summary of the Invention

[0005] This invention provides a file splitting method and apparatus that can accelerate file splitting and reduce time and labor costs.

[0006] In a first aspect, embodiments of the present invention provide a file splitting method applicable to a distributed cluster; multiple servers in the distributed cluster are used to execute their respective splitting tasks; the method includes: for any one of the multiple servers, the server obtains a shard to be split from the file to be split according to the shard position in the splitting task; the server determines the category corresponding to each line of data in the shard to be split according to the business logic in the file processing configuration and records it in the local variable of the category; the server writes the local variable of each category into each category file in a locked manner according to the category storage information in the file processing configuration.

[0007] Using the above method, multiple servers in the distributed cluster each obtain the corresponding shard to be split from the file to be split according to the shard position in the splitting task. This allows multiple servers in the distributed cluster to split each shard of the file to be split independently, achieving multi-threaded splitting processing and accelerating the splitting speed. The shard to be split is then split according to the business logic in the file processing configuration; this allows the business logic to be input by the user, increasing the flexibility and variability of the splitting method. Each line of data in the shard to be split is recorded in a local variable of the category to which that line belongs. By setting local variables, the shard to be split can be obtained through memory mapping, ensuring that each server in the cluster only has the content of the shard currently being split, without needing to store the entire file to be split or all shards split by that server, reducing resource consumption and ensuring the server's performance remains optimal. Furthermore, setting local variables eliminates the need to generate files for each category, reducing the server's workload and further guaranteeing performance. The server writes local variables into the corresponding category files based on the category storage information in the file processing configuration. Since the category storage information in the file processing configuration can be entered by the user, the category information can be adjusted at any time according to the user's needs, increasing the flexibility of the classification.

[0008] In one possible design, before recording to the local variables of the classification, the following is also included:

[0009] The server obtains the assembly data corresponding to each line of data according to the assembly rules of each category in the file processing configuration, and assembles the data according to the business logic.

[0010] By using the above method, and setting various category assembly rules in the file processing device, the server can obtain the assembly data corresponding to each line of data according to these rules. Furthermore, based on business logic, the content to be split and the assembly data are assembled to obtain a combined file of the file to be split and the assembly data. In this way, while classifying, the user can add content to the file to be split without requiring the user to manually assemble the file to be split and the assembly data. This allows for adding file content while splitting the file, accelerating file processing speed and saving time and manpower costs.

[0011] In one possible design, local variables for each category are written to each category file in a locked manner, including: for any category's local variables, the server acquires a write lock for that category in the Redis server; the server writes the local variables for that category to the corresponding category folder; and the server releases the write lock for that category in the Redis server.

[0012] Using the above method, when writing local variables for any category to their respective category files, the server acquires the write lock corresponding to that category in the Redis server, and then writes the local variables for that category to the corresponding category folder. This ensures that for a given category folder, only one server in the distributed cluster performs the write operation, preventing the content from becoming corrupted and causing loss of content in the files to be split due to multiple servers writing to the same category folder simultaneously, thus guaranteeing the accuracy of the written content. Finally, after the write operation is complete, the write lock for that category is released in the Redis server, ensuring that other servers can write the content corresponding to that category to the category file.

[0013] In one possible design, the server obtains the fragment to be split from the file to be split based on the fragment position in the splitting task, including: the server determining the read attribute of the fragment to be split; the read attribute being read as a whole fragment or read at a set size; the server obtaining the fragment to be split from the file to be split according to the read attribute and the fragment position in the splitting task.

[0014] Using the above method, the server can determine whether a fragment to be split is read as a whole or a fragment of a set size based on its read attributes. This ensures that when the read attribute of a fragment is "read as a whole," the server retrieves the entire fragment from the file to be split based on its position in the splitting task, guaranteeing the integrity of the fragment's content. When the read attribute of a fragment is "read as a set size," the server retrieves the fragment of that size from the file to be split based on its position in the splitting task, minimizing server resource consumption, ensuring server performance, accelerating the splitting process, and maintaining the integrity of the fragment's content.

[0015] In one possible design, the read attribute is a read of a set size, including:

[0016] The set size can be determined by the following formula:

[0017] V = MIN(V, G / R)

[0018] Where V is the set size; G is the maximum content read each time, which is determined by historical experience values ​​and the performance of the server; and R is the size of each line of content.

[0019] Using the above method, based on historical experience and server performance, the maximum amount of content that can be read each time and the size of each line of content are determined. The maximum number of lines of data that can be retrieved each time is determined by the ratio, and a minimum value is determined between the set size and the maximum number of lines of data that can be retrieved each time. In this way, it can be ensured that the content to be split into fragments is retrieved according to the minimum number of lines each time, ensuring that the server performance is always in the optimal state and speeding up the server processing speed.

[0020] In one possible design, the method includes: storing the classification files in each data center; after writing the local variables of each classification into each classification file in a locked manner, the method further includes: the data center shards the classification files of the data center, and sorts each shard using a heap sort method to obtain sorted classification files.

[0021] By using the above method, classifying and sorting the classified files in the data center, we can quickly obtain ordered classified files according to the corresponding sorting rules, thereby increasing the orderliness of the classified files.

[0022] In one possible design, before writing the local variables of each category into each category file in a locked manner, the method further includes: the server opening each category file according to the file processing configuration; after writing the local variables of each category into each category file in a locked manner, the method further includes: the server closing each category file according to the file processing configuration after determining that all category files have been written; the file processing configuration is obtained by writing on a preset template; the preset template is provided with functional code for opening files, reading files, and closing files.

[0023] Using the above method, since the preset template includes code for opening, reading, and closing files, users are not required to write additional code, reducing their workload and time costs. Furthermore, given the complexity of this code, this method not only reduces user workload and time costs but also eliminates the possibility of file corruption due to errors in writing the file opening or reading code. It also eliminates the situation where files waiting to be processed continuously consume server resources due to users forgetting to write or incorrectly writing the file closing code.

[0024] Secondly, embodiments of the present invention provide a file splitting apparatus suitable for a distributed cluster; multiple servers in the distributed cluster are used to perform their respective splitting tasks; the apparatus includes:

[0025] The acquisition module, for any one of the plurality of servers, is used to acquire the shard to be split from the file to be split based on the shard position in the splitting task;

[0026] The processing module is used to determine the category corresponding to each line of data in the shard to be split according to the business logic in the file processing configuration and record it in the local variable of the category;

[0027] The processing module is also used to write local variables of each category into each category file in a locked manner according to the category storage information in the file processing configuration.

[0028] Thirdly, embodiments of the present invention also provide a computing device, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory and executing the method described in various possible designs of the first aspect according to the obtained program.

[0029] Fourthly, embodiments of the present invention also provide a computer-readable non-volatile storage medium including computer-readable instructions that, when read and executed by a computer, cause the computer to perform the method described in various possible designs of the first aspect.

[0030] These or other implementations of the present invention will become more readily apparent in the following description of the embodiments. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A schematic diagram of the architecture of a file splitting method provided in an embodiment of the present invention;

[0033] Figure 2 A flowchart illustrating a file splitting method provided in an embodiment of the present invention;

[0034] Figure 3 A flowchart illustrating a file splitting method provided in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of a file splitting device provided in an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0037] Figure 1 This invention provides a file splitting system architecture suitable for distributed clusters; multiple servers in the distributed cluster are used to execute their respective splitting tasks. Figure 1 Server cluster 103 can be used to split the file to be split; server cluster 105 can be used to store the categorized files after splitting the file to be split; server cluster 106 can be used to store categorized files of a certain category; server 101 can be a server storing the file to be split; and server 104 can be a Redis server. For ease of description, server cluster 103 is referred to as the splitting server cluster, server cluster 105 as the data center cluster, server 101 as the file server, and server 104 as the Redis server. Thus, after the splitting server cluster 103 obtains the file to be split from the file server 101, any one of the splitting servers in the splitting server cluster 103 splits the file to be split into shards and generates various splitting tasks. Each splitting server in the splitting server cluster 103 can then obtain the shards to be split according to the splitting tasks and split the shards to obtain local variables for each category. Furthermore, the local variables for each category are written into the corresponding data center cluster 106 in the data center cluster 105. Subsequently, the data center clusters 106 of each category in the data center cluster 105 can further process the categorized files. In the above process, the server cluster 103 to be split can also obtain the files to be split locally, and the specific method of obtaining the files to be split is not limited. The data center cluster 106 consists of at least one data center server. The Redis server 104 can be used to store each splitting task, so that each splitting server in the splitting server cluster 103 can obtain the shards to be split according to each splitting task; due to the high concurrency availability and atomicity of the Redis server, the speed, accuracy and integrity of the splitting process of the files to be split can be guaranteed.

[0038] Based on this, embodiments of the present invention provide a flow chart for a file splitting method, such as... Figure 2 As shown, including:

[0039] Step 201: For any one of the plurality of servers, the server obtains the shard to be split from the file to be split according to the shard position in the splitting task;

[0040] Here, as Figure 1 In this architecture, for any one of the split servers in the split server cluster 103, the split server retrieves the fragment to be split from the file to be split locally on file server 101 or any one or more split servers in the split server cluster 103, based on the fragment position in the split task. For example, if the split server records the start position of the fragment to be split as the 501st byte and the end position as the 600th byte of the file to be split in the fragment position obtained from the split task, then the split server retrieves bytes 501-600 of the file to be split.

[0041] Step 202: The server determines the category corresponding to each row of data in the shard to be split according to the business logic in the file processing configuration and records it in the local variable of the category;

[0042] Here, in the previous example, if the file to be split contains financial information of Shanghai customers, Beijing customers, and Guangzhou customers, then after the splitting server obtains the corresponding splitting task's shard to be split, that is, bytes 501-600 of the file to be split, according to the business logic in the file processing configuration, it records the row data belonging to Shanghai customers in the local variable of Shanghai customers, the row data belonging to Beijing customers in the local variable of Beijing customers, and the row data belonging to Guangzhou customers in the local variable of Guangzhou customers.

[0043] The file processing configuration here can also include a list of data centers, such as... Figure 1 Each data center cluster (106) corresponds to a type of category file. Within this data center list information, a configuration file is created for each data center cluster corresponding to each category file; that is, a context is created. The context can be stored in key-value format. The category file information created for each data center cluster is placed into the context, where the key is the data center cluster identifier and the value is the category file information.

[0044] Step 203: The server writes the local variables of each category into each category file in a locked manner according to the category storage information in the file processing configuration.

[0045] Here, the categorized storage information can be the information about the categorized file corresponding to each data center cluster in the data center list information in step 202, as well as the storage path of the categorized file in the corresponding data center cluster. This allows each split server to write the local variables of each category into the categorized file in the corresponding data center cluster. The writing process is performed using locking to prevent multiple split servers from writing to the same categorized file simultaneously, which could cause the categorized file content to become corrupted.

[0046] Using the above method, multiple servers in the distributed cluster each obtain the corresponding shard to be split from the file to be split according to the shard position in the splitting task. This allows multiple servers in the distributed cluster to split each shard of the file to be split independently, achieving multi-threaded splitting processing and accelerating the splitting speed. The shard to be split is then split according to the business logic in the file processing configuration; this allows the business logic to be input by the user, increasing the flexibility and variability of the splitting method. Each line of data in the shard to be split is recorded in a local variable of the category to which that line belongs. By setting local variables, the shard to be split can be obtained through memory mapping, ensuring that each server in the cluster only has the content of the shard currently being split, without needing to store the entire file to be split or all shards split by that server, reducing resource consumption and ensuring the server's performance remains optimal. Furthermore, setting local variables eliminates the need to generate files for each category, reducing the server's workload and further guaranteeing performance. The server writes local variables into the corresponding category files based on the category storage information in the file processing configuration. Since the category storage information in the file processing configuration can be entered by the user, the category information can be adjusted at any time according to the user's needs, increasing the flexibility of the classification.

[0047] This application provides a file splitting method, which further includes, before recording the data into the local variables of the category: the server obtains the assembly data corresponding to each line of data according to the assembly rules of each category in the file processing configuration, and assembles the data according to the business logic.

[0048] In the previous example, the file to be split contained financial information for customers in Shanghai, Beijing, and Guangzhou, but not their identity information, such as ID card numbers or mobile phone numbers. When it is necessary to assemble the assembled data, such as ID card numbers or mobile phone numbers, into the file to be split, the splitting server, according to the business logic in the file processing configuration, writes the assembled data, such as ID card numbers or mobile phone numbers, of each customer into the corresponding position in the row data of that customer's financial information, so that the file to be split contains the customer's identity information.

[0049] Here, the assembled data can be written into the context in step 202, which will facilitate the subsequent writing of each category file into the data center cluster. This context can be shared in all threads of the split server cluster 103.

[0050] This application provides a method for writing local variables of each category into separate category files after file splitting. Specifically, local variables of each category are written to their respective category files using a locking mechanism. This includes: for any category's local variables, the server acquires a write lock for that category in a Redis server; the server writes the local variables of that category to the corresponding category folder; and the server releases the write lock for that category in the Redis server. In other words, before a splitting server writes the local variables of a category into the category file of the corresponding data center cluster, it acquires a write lock for that category from the Redis server, and then writes the local variables of that category into the category file of the corresponding data center cluster. This prevents the category file from being written to by multiple splitting servers simultaneously, which could cause content corruption. After writing is complete, the write lock for that category is released in the Redis server, allowing other splitting servers to write the local variables of that category. Here, the splitting server that opens the category file in the data center cluster and the splitting server that acquires the write lock can be different.

[0051] This application provides a method for obtaining a fragment to be split, wherein the server obtains the fragment to be split from the file to be split according to the fragment position in the splitting task, including: the server determining the read attribute of the fragment to be split; the read attribute being whole fragment read or set size read; the server obtaining the fragment to be split from the file to be split according to the read attribute and the fragment position in the splitting task.

[0052] Here, in the previous example, if the splitting server determines that the read attribute of the fragment to be split is a whole fragment read, then bytes 501-600 are read into the cache and split. If the size is set to 20 bytes, then only 20 bytes are cached each time and split. Here, to ensure the integrity of the content read each time, the size is set to obtain at least one whole row of data.

[0053] This application provides another method for obtaining fragments to be split, wherein the reading attribute is a set size reading, including: the set size can be determined by the following formula:

[0054] V = MIN(V, G / R)

[0055] Where V is the set size; G is the maximum content read each time, which is determined by historical experience values ​​and the performance of the server; and R is the size of each line of content.

[0056] Here, if the size is set to the number of rows, the above formula can ensure that the number of rows of data retrieved by the splitting server each time does not exceed G / R, because G can be the maximum content that the splitting server can read when the splitting process is in an optimal state; therefore, it can be ensured that the splitting server has the best performance when splitting the fragments to be split, thus speeding up the splitting speed of the files to be split.

[0057] Based on the above-described splitting process, this application also provides a method for reading fragments to be split, as shown below:

[0058] The splitting server maps the filePair(k) to be split into a cache via memory mapping. filePair(k) represents the k-th fragment in the filePairList to be split. When the read content reaches a set size (hereinafter referred to as V), the parameter isPartHandle is determined to be true, indicating that the read attribute of the fragment to be split is partial reading. The corresponding business logic method is then called to perform the splitting.

[0059] local rowList; / / Collection of row data to be split into shards.

[0060] local row; / / The content of each row of data read.

[0061] local count = 0;

[0062] for bytedata in filePair(k).data do / / bytedata is the content of the filePair(k) to be split.

[0063] if bytedata == '\n' / / Determine if the last byte of the set size is a newline character.

[0064] rowList.add(row); / / If it is a newline character, it means that a row has been read and the read row content is added to rowList.

[0065] row.clear(); / / Clears the row, preparing to read the next row of data.

[0066] count++; / / Increment the number of rows read by 1.

[0067] if count==V and isPartHandle=true / / Determine if the total number of rows read this time reaches the value V, and the read attribute of the partition to be split is that it does not need to be processed as a whole.

[0068] callBusinessSplitor(rowList); / / Read the size of the V value and call the corresponding business logic to split the file.

[0069] rowList.clear(); / / Clears the rows that have been read.

[0070] count = 0; / / Clear count to zero and start counting again.

[0071] End.

[0072] Else.

[0073] row.append(data); / / If the value of V read is not the end of the line, continue counting downwards until the first newline character is obtained, and then add the content counted downwards to the line content.

[0074] End.

[0075] End.

[0076] Here, if isPartHandle = false, it means that the read attribute of the fragment to be split is read as a whole fragment.

[0077] This application provides another method for sorting classification files. The method includes: storing each classification file in each data center; writing the local variables of each classification into each classification file in a locked manner; and further including: the data center sharding the classification files of the data center and sorting each shard using a heap sort method to obtain the sorted classification files.

[0078] Here, after the data center cluster obtains the categorized file, it can divide the file into categorized fragments. These fragments are then sorted using a pre-defined or input sorting rule to obtain an ordered categorized file. For example, if the data center cluster obtains a categorized file and divides it into 20 fragments, and if there are 10 servers in the cluster, each server receives 2 fragments. Each server then sorts these 2 fragments according to the sorting rule to obtain an ordered fragment. This process is repeated, with each of the 10 servers sorting every 2 fragments from the initial 20 to obtain 10 ordered fragments. These 10 ordered fragments can then be distributed among 5 servers, which continue sorting them according to the same rule until a complete ordered categorized file is obtained. The sorting rule can be based on content size, number of keywords, etc., and can be configured as needed; the specific sorting rule is not limited.

[0079] This application provides another method for splitting files. Before writing the local variables of each category into each category file in a locked manner, the method further includes: the server opening each category file according to the file processing configuration; after writing the local variables of each category into each category file in a locked manner, the method further includes: the server closing each category file according to the file processing configuration after determining that all category files have been written; the file processing configuration is obtained by writing on a preset template; the preset template is provided with functional codes for opening, reading, and closing files. In other words, this application can set a preset template and include functional codes for opening, reading, and closing files within it. This eliminates the need for users to write additional functional codes, reducing user workload and time costs. Furthermore, it solves the problem of high user workload and long writing time due to the complexity of the functional codes; it eliminates the possibility of file corruption due to user errors in writing the function codes for opening or reading files; and it eliminates the situation where files to be processed continuously occupy server resources due to users forgetting to write or writing incorrect function codes for closing files.

[0080] Based on the above process, embodiments of the present invention provide a process for a file splitting method, such as... Figure 3 As shown, it includes:

[0081] Step 301: The split server obtains the file to be split and splits the file into fragments to generate a splitting task.

[0082] Step 302: The split server sends each split task to the Redis server.

[0083] Step 303: Each split server receives the broadcast from the Redis server and obtains the splitting task from the Redis server.

[0084] Step 304: Each splitting server obtains the shard to be split according to the shard position in its respective splitting task. If there are assembly rules for each category in the file processing configuration, then the assembly data corresponding to each row of data is obtained.

[0085] Step 305: Each splitting server classifies and assembles the data and assembly data in the shards to be split, and records them in the local variables of each category.

[0086] Step 306: Each split server writes the local variables of each category into the category file in the corresponding data center cluster.

[0087] Step 307: The data center cluster sorts its respective classification files to obtain an ordered classification file.

[0088] Based on the same concept, embodiments of the present invention provide a file splitting device. Figure 4 A schematic diagram of a file splitting device provided in an embodiment of the present invention is shown below. Figure 4 The following are examples:

[0089] The acquisition module 401, for any one of the plurality of servers, is used to acquire the shard to be split from the file to be split according to the shard position in the splitting task;

[0090] The processing module 402 is used to determine the category corresponding to each row of data in the fragment to be split according to the business logic in the file processing configuration and record it in the local variable of the category;

[0091] The processing module 402 is further configured to write local variables of each category into each category file in a locked manner according to the category storage information in the file processing configuration.

[0092] In one possible design, the acquisition module 401 is further configured to: acquire the assembly data corresponding to each row of data according to the assembly rules of each category in the file processing configuration, and assemble the data according to the business logic.

[0093] In one possible design, the processing module 402 is specifically used for: for any category's local variable, the server acquires a write lock for that category in the Redis server; the server writes the local variable of that category to the corresponding category folder; and the server releases the write lock for that category in the Redis server.

[0094] In one possible design, the acquisition module 401 is specifically used for: the server determining the read attribute of the fragment to be split; the read attribute being whole fragment read or read at a set size; the server acquiring the fragment to be split from the file to be split according to the read attribute and the fragment position in the splitting task.

[0095] In one possible design, the read attribute is a read of a set size, including:

[0096] The set size can be determined by the following formula:

[0097] V = MIN(V, G / R)

[0098] Where V is the set size; G is the maximum content read each time, which is determined by historical experience values ​​and the performance of the server; and R is the size of each line of content.

[0099] In one possible design, the processing module 402 is specifically used for: storing the classification files in each data center; after writing the local variables of each classification into each classification file in a locked manner, it is also used for: the data center sharding the classification files of the data center, sorting each shard using a heap sort method, and obtaining the sorted classification files.

[0100] In one possible design, the processing module 402 is further configured to: open each category file according to the file processing configuration; after writing the local variables of each category into each category file in a locked manner, the server further includes: after determining that all category files have been written, closing each category file according to the file processing configuration; the file processing configuration is obtained by writing on a preset template; the preset template is provided with functional code for opening files, reading files, and closing files.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A file splitting method, characterized in that, Suitable for distributed clusters; The distributed cluster contains multiple servers that execute their respective splitting tasks; the method includes: For any one of the plurality of servers, the server obtains the shard to be split from the file to be split according to the shard position in the splitting task; The server determines the category corresponding to each row of data in the shard to be split according to the business logic in the file processing configuration and records it in the local variable of the category; based on the local variable, the shard to be split is obtained through memory mapping. The server writes the local variables of each category into each category file in a locked manner according to the category storage information in the file processing configuration; the category files are stored in each data center. The data center divides the data center's classification files into fragments, and sorts each fragment using a heap sort method to obtain the sorted classification files.

2. The method as described in claim 1, characterized in that, Before recording it in the local variables of the classification, it also includes: The server obtains the assembly data corresponding to each line of data according to the assembly rules of each category in the file processing configuration, and assembles the data according to the business logic.

3. The method as described in claim 1, characterized in that, Local variables for each category are written to their respective category files in a locked manner, including: For any local variable of any category, the server acquires a write lock for that category in the Redis server; The server writes the local variables of the category into the corresponding category folder; The server releases the write lock for the category in the Redis server.

4. The method as described in claim 1, characterized in that, The server obtains the fragments to be split from the file to be split based on the fragment position in the splitting task, including: The server determines the read attributes of the fragment to be split; the read attributes are either reading the entire fragment or reading a specified size; The server retrieves the fragments to be split from the file to be split according to the read attributes and the fragment positions in the splitting task.

5. The method as described in claim 4, characterized in that, The read attribute is for reading at a set size, including: The set size can be determined by the following formula: V = MIN(V, G / R) Where V is the set size; G is the maximum content read each time, which is determined by historical experience values ​​and the performance of the server; and R is the size of each line of content.

6. The method as described in claim 1, characterized in that, Before writing the local variables of each category to the respective category file in a locked manner, the following steps are also included: The server opens the files of each category according to the file processing configuration; After writing the local variables of each category into the respective category files in a locked manner, it also includes: After determining that all categorized files have been written, the server closes each categorized file according to the file processing configuration; the file processing configuration is obtained by writing on a preset template; the preset template is set with functional code for opening files, reading files, and closing files.

7. A file splitting device, characterized in that, Suitable for distributed clusters; The distributed cluster contains multiple servers that execute their respective splitting tasks; the apparatus includes: The acquisition module, for any one of the plurality of servers, is used to acquire the shard to be split from the file to be split based on the shard position in the splitting task; The processing module is used to determine the category corresponding to each line of data in the shard to be split according to the business logic in the file processing configuration and record it in the local variable of the category; based on the local variable, the shard to be split is obtained through memory mapping. The processing module is further configured to write local variables of each category into each category file in a locked manner according to the category storage information in the file processing configuration; the category files are stored in each data center. The data center divides the data center's classification files into fragments, and sorts each fragment using a heap sort method to obtain the sorted classification files.

8. A computing device, characterized in that, include: memory for storing computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method according to any one of claims 1 to 6 in accordance with the obtained program.

9. A computer-readable non-volatile storage medium, characterized in that, Includes a computer-readable program that, when read and executed by a computer, causes the computer to perform the method as described in any one of claims 1 to 6.

Images